AltCLIP

AltCLIP is a bilingual (Chinese-English) multimodal representation model that extends OpenAI's CLIP by replacing its text encoder with XLM-R, enabling strong zero-shot image classification and cross-modal retrieval in both languages. The model uses a two-stage training process: first, parallel knowledge distillation (teacher learning) on large-scale text corpora, then bilingual contrastive learning on approximately 2 million Chinese-English image-text pairs. Both AltCLIP and its 9-language variant AltCLIP-m9 (supporting English, Chinese, Spanish, French, Russian, Japanese, Korean, Arabic, and Italian) use a ViT-L image encoder and have a model size of 3.22 GB. On the Flickr30k benchmark, AltCLIP achieves the following retrieval performance (R@1/R@5/R@10):

English

Task	R@1	R@5	R@10
Text-to-Image	66.3	87.8	92.7
Image-to-Text	85.9	97.7	99.1

Chinese

Task	R@1	R@5	R@10
Text-to-Image	63.7	86.3	92.1
Image-to-Text	84.7	97.4	98.7

The enhanced variant AltCLIP* (trained with additional data) sets new state-of-the-art results on ImageNet-CN, Flickr30k-CN, COCO-CN, and XTD, achieving a mean recall of 90.4 on English and 89.2 on Chinese tasks. AltCLIP also serves as the text-image backbone for the AltDiffusion text-to-image generation model.