Fine-Grained Image-Text Correspondence with
Cost Aggregation for Open-Vocabulary Part Segmentation

Jiho Choi^{1 *}, Seonho Lee^{1 *}, Seungho Lee², Minhyun Lee², Hyunjung Shim¹
(* indicates equal contributions)
¹Graduate School of Artificial Intelligence, KAIST, Republic of Korea
²School of Integrated Technology, Yonsei University, Republic of Korea
{jihochoi, glanceyes, kateshim}@kaist.ac.kr, {seungholee, lmh315}@yonsei.ac.kr

---

PartCATSeg

PartCATSeg is a novel framework to address the critical challenges of OVPS(Open-Vocabulary Part Segmentation) and significantly improve model performance by leveraging the cost volume.

Updates

• [2025.04.15] 👨‍💻 The official codes have been released!
• [2025.02.26] 🎉 Our paper has been accepted to CVPR 2025!
• [2025.01.16] 📄 Our paper is now available! You can find the paper here.

Installation

# ------------------
#     Init conda
# ------------------
conda create --name partcatseg python=3.8 -y
conda activate partcatseg
pip install --upgrade pip
conda install cuda=12.4.1 -c nvidia -y
pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
pip install timm==0.9.1
pip install scikit-image==0.21.0
pip install scikit-learn==0.24.2
pip install opencv-python==4.5.5.64
pip install hydra-core==1.3.2
pip install openmim==0.3.6
pip install mmsegmentation==0.29.1
pip install tokenizers==0.11.1
pip install Pillow~=9.5
pip install numpy==1.23.0
pip install einops ftfy regex fire ninja psutil gdown

# --------------------------
#     Install Detectron2
# --------------------------
pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -c "import detectron2; print(detectron2.__version__)"  # 0.6

# --------------------------
#     Install mmcv
# --------------------------
# pip install mmcv-full==1.7.1
# => if an error occurs
pip install mmcv-full==1.7.1 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.html
python -c "import mmcv; print(mmcv.__version__)"  # 1.7.1

Prepare Datasets

cd datasets

PascalPart116

• You can find further information in the OV-PARTS GitHub repository.

gdown https://drive.google.com/uc?id=1QF0BglrcC0teKqx15vP8qJNakGgCWaEH
tar -xzf PascalPart116.tar.gz
find datasets/PascalPart116/images/val/ -name '._*' -delete
find datasets/PascalPart116/ -name '._*' -delete

ADE20KPart234

gdown https://drive.google.com/uc?id=1EBVPW_tqzBOQ_DC6yLcouyxR7WrctRKi
tar -xzf ADE20KPart234.tar.gz

PartImageNet (Seg)

• Download the LOC_synset_mapping.txt file from this link and place it in the datasets folder.
• Download PartImageNet_Seg from PartImageNet and extract it into the datasets folder.

PartImageNet (OOD)

• TBA

Preprocess Datasets

• PascalPart116
• ADE20KPart234
• PartImageNet (Seg)
• PartImageNet (OOD) - TBA

# PascalPart116
python baselines/data/datasets/mask_cls_collect.py \
    datasets/PascalPart116/annotations_detectron2_part/val \
    datasets/PascalPart116/annotations_detectron2_part/val_part_label_count.json

python baselines/data/datasets/mask_cls_collect.py \
    datasets/PascalPart116/annotations_detectron2_obj/val \
    datasets/PascalPart116/annotations_detectron2_part/val_obj_label_count.json

# ADE20KPart234
# (no preprocessing required)

# PartImageNet (Seg)
cd datasets
python partimagenet_preprocess.py --data_dir PartImageNet
# Make sure to have LOC_synset_mapping.txt in the datasets folder mentioned above.

# PartImageNet (OOD)
# TBA

Pre-trained Weights

Please first donwload the pre-trained CAT-Seg weight model_final_base.pth from the following link to use them in the training.

Model	Checkpoint
CAT-Seg	download link

Make sure to place the downloaded weights in the pretrain_weights folder.

For the evaluation, we provide the pre-trained weights for the PartCATSeg model on the following datasets:

Model	Setting	Dataset	Checkpoint
PartCATSeg	zero-shot	Pascal-Part-116	model
PartCATSeg	zero-shot	ADE20K-Part-234	model
PartCATSeg	zero-shot	PartImageNet (Seg)	model
PartCATSeg	zero-shot	PartImageNet (OOD)	TBA

mkdir weights && cd weights

# Pascal-Part-116
# PartCATSeg (partcatseg_voc.pth (927.9M))
gdown https://drive.google.com/uc?id=1WhbMVGKm7KkHacz9rY5vU7oVILzvI_OK

# ADE20K-Part-234
# PartCATSeg (partcatseg_ade.pth (927.9M))
gdown https://drive.google.com/uc?id=12JgECVJco81EoI_3ZR93S2z7zdvwAyU9

# PartImageNet (Seg)
# PartCATSeg (partcatseg_partimagenet.pth (927.9M))
gdown https://drive.google.com/uc?id=1Jp8IjrTlQKGUOaSU8REvB4wvHmyNb2yV

Usage (Run)

Zero-Shot Prediction

# -------------
#     Train
# -------------
python train_net.py \
    --num-gpus 4 \
    --config-file configs/zero_shot/partcatseg_voc.yaml

# -----------------
#     Inference
# -----------------
python train_net.py \
    --num-gpus 4 \
    --config-file configs/zero_shot/partcatseg_voc.yaml \
    --eval-only MODEL.WEIGHTS ./weights/partcatseg_voc.pth

Project Structure

[PROJECT_ROOT]
├── datasets/
│   ├── PascalPart116/
│   │   ├── images
│   │   ├── annotations_detectron2_obj
│   │   └── annotations_detectron2_part
│   ├── ADE20KPart234/
│   └── PartImageNet/
├── weights/
│   ├── partcatseg_voc.pth
│   └── partcatseg_ade.pth
├── configs/
│   ├── zero_shot
│   │   ├── partcatseg_voc.yaml
│   │   ├── partcatseg_ade.yaml
│   │   ├── partcatseg_partimagenet.yaml
│   │   └── ...
├── baselines/
│   ├── evaluation/
│   │   └── partcatseg_evaluation.py
│   ├── data/
│   │   ├── dataset_mappers/
│   │   │   └── object_part_mapper.py
│   │   └── datasets/
│   │       └── register_pascal_part_116.py
│   ├── modeling/
│   │   ├── backbone/
│   │   │   └── dinov2_backbone.py
│   │   ├── heads/
│   │   │   └── part_cat_seg_head.py
│   │   └── transformer/
│   │       ├── part_cat_seg_model.py
│   │       ├── part_cat_seg_predictor.py
│   │       └── part_cat_seg_model.py
│   ├── utils/
│   │   └── visualizer.py
│   ├── third_party/
│   ├── partcatseg.py
│   └── config.py
├── README.md
└── train_net.py

Acknowledgement

We would like to express our gratitude to the open-source projects and their contributors, including PartCLIPSeg, OV-PARTS, CLIPSeg, Mask2Former, CLIP, and OV-DETR.