TESACLIP

Official implementation of TESACLIP: Text-Enhanced Semantic Alignment for Training-Free Open-Vocabulary Segmentation.

Qingwu Duan and Qiaoling Zhou
ShanghaiTech University

TESACLIP improves frozen CLIP representations at inference time. It requires no training, parameter updates, extra training data, or auxiliary vision foundation model.

Paper and arXiv links will be added when the preprint is public.

Repository: https://github.com/Fourques/TESACLIP

Method

TESACLIP addresses two limitations of training-free open-vocabulary semantic segmentation:

1. Inter-class text confusion. Domain prompts, contrastive text enhancement, and soft text-feature decorrelation separate confusable class representations.
2. Text-patch alignment failure. Partial encoder restoration and image-conditioned prototypes improve dense matching, while background logit calibration removes the scoring advantage of composite background classes.

All operations use frozen CLIP features and add no learned parameters.

Results

The table reports mIoU with CLIP ViT-B/16 and PAMR, using the configurations included in this repository.

Method	Cityscapes	VOC20	VOC21	Context59	Context60	ADE20K	COCO-Object	COCO-Stuff	Avg.
NACLIP	38.27	83.03	64.07	38.35	34.99	19.05	36.19	25.69	42.45
TESACLIP	41.10	84.43	66.51	40.99	36.21	20.66	37.16	26.25	44.16

Results without PAMR and comparisons under different auxiliary-model settings are reported in the paper. Please compare methods only under compatible backbones, input resolutions, and post-processing protocols.

Installation

The reference environment uses Python 3.9, PyTorch 1.13.1, CUDA 11.7, MMCV 2.0.1, MMEngine 0.8.4, and MMSegmentation 1.1.1.

conda create -n tesaclip python=3.9 -y
conda activate tesaclip

pip install torch==1.13.1 torchvision==0.14.1 \
  --extra-index-url https://download.pytorch.org/whl/cu117
pip install -U openmim
mim install "mmcv==2.0.1"
pip install "mmengine==0.8.4" "mmsegmentation==1.1.1"
pip install -r requirements.txt

Alternatively, environment.yml installs the Python and PyTorch base; run the three OpenMMLab installation commands above afterward.

The first run downloads the OpenAI CLIP ViT-B/16 checkpoint to the standard CLIP cache. In an offline environment, download the checkpoint separately and pass its local path with --backbone.

Data preparation

Obtain each dataset from its original provider and arrange it as follows:

data/
├── cityscapes/
│   ├── leftImg8bit/val/
│   └── gtFine/val/
├── VOCdevkit/
│   ├── VOC2012/
│   │   ├── JPEGImages/
│   │   └── SegmentationClass/
│   └── VOC2010/
│       ├── JPEGImages/
│       └── SegmentationClassContext/
├── ADEChallengeData2016/
│   ├── images/validation/
│   └── annotations/validation/
└── coco_stuff164k/
    ├── images/val2017/
    └── annotations/val2017/

Create the COCO-Object annotations once before evaluation:

python datasets/cvt_coco_object.py data/coco_stuff164k

Datasets are not distributed with this repository. Their original licenses and terms apply.

Evaluation

Evaluate one benchmark:

python eval.py \
  --config configs/cfg_city_scapes_tesa.py \
  --data-root data/cityscapes \
  --pamr on

Evaluate all configured benchmarks after preparing their expected data roots:

for config in configs/cfg_*_tesa.py; do
  python eval.py --config "$config" --pamr on
done

Use --pamr off for the no-post-processing protocol. Logs are written to work_logs/ by default.

Configuration

The released config files reproduce the reported per-dataset settings. The main parameters are:

Parameter	Role
cte_alpha	Contrastive text enhancement weight
tfd_beta	Text-feature decorrelation strength
ssca_temp, ssca_lambda	Image-conditioned prototype temperature and blend
residual_alpha, ffn_alpha	Partial encoder restoration weights
lla_tau, lla_mode	Background logit calibration

Components that are incompatible with composite background labels or very large label sets are disabled in the corresponding configs, as described in the paper.

Repository structure

TESACLIP/
├── tesa.py              # Model and inference-time refinement
├── eval.py              # Evaluation CLI
├── pamr.py              # Optional PAMR post-processing
├── datasets.py          # MMSeg dataset registrations
├── clip/                # Modified OpenAI CLIP implementation
├── configs/             # Eight benchmark configurations
├── prompts/             # Prompt banks and contrastive descriptions
├── datasets/            # Dataset conversion utility
└── assets/              # README figures

Reproducibility notes

• Main experiments use a frozen CLIP ViT-B/16 backbone.
• Cityscapes uses a short side of 560 pixels; other sliding-window benchmarks use 336 pixels. VOC20 and VOC21 use full-image inference.
• Sliding-window crops are 224 by 224 with stride 112.
• PAMR uses 10 iterations with dilations 8 and 16.
• No dataset annotations or pretrained weights are committed to this repo.

Citation

The arXiv BibTeX entry will be added after the identifier is assigned. GitHub also exposes the current author metadata from CITATION.cff.

Acknowledgements and license

TESACLIP builds on NACLIP, OpenAI CLIP, and PAMR. See THIRD_PARTY_NOTICES.md for details.

The original TESACLIP code is released under the MIT License. Third-party components retain their respective licenses.