CLIP ViT-B/32

CLIP (Contrastive Language-Image Pre-Training) is a zero-shot image classification model that learns visual concepts from natural language supervision, enabling classification of images into arbitrary categories without task-specific training data.

Architecture

The model uses a ViT-B/32 Vision Transformer as the image encoder and a masked self-attention Transformer as the text encoder. Both encoders are jointly trained via a contrastive loss to maximize the similarity of correct (image, text) pairs. The resulting representations allow the model to perform zero-shot classification by computing the similarity between an image and a set of candidate text descriptions.

Training Data

CLIP was trained on 400 million (image, text) pairs collected from publicly available internet sources, including crawled websites and existing datasets such as YFCC100M. This large-scale, diverse dataset supports strong generalization across many visual domains.

Performance

On ImageNet, CLIP matches the accuracy of the original ResNet-50 in a zero-shot setting without using any of the 1.28 million labeled training examples. The model has been evaluated on over 30 computer vision benchmarks spanning OCR, action recognition, geo-localization, fine-grained classification, and more. Key results include:

• Gender classification on FairFace: >96% accuracy across all race groups, with Middle Eastern highest (98.4%) and White lowest (96.5%).
• Racial classification on FairFace: ~93% average accuracy.
• Age classification on FairFace: ~63% average accuracy.

Limitations and Bias

CLIP struggles with fine-grained classification, object counting, and tasks requiring precise spatial reasoning. Performance varies significantly with class taxonomy design, and the model exhibits racial and gender biases—for example, disparities in crime-related and non-human animal classification across demographics. These limitations should be carefully evaluated before any deployment.