Jina CLIP V1

jinaai/jina-clip-v1 is an English multimodal text-image embedding model that achieves state-of-the-art performance on both text-image retrieval and text-text retrieval tasks within a single model, bridging the gap between dedicated text embedding models and CLIP-style cross-modal models.

Dual-Modal Capabilities

Traditional text embedding models excel at text-to-text retrieval but cannot handle cross-modal queries. Standard CLIP models effectively align image and text embeddings but are not optimized for text-to-text retrieval. jina-clip-v1 matches the text retrieval efficiency of jina-embeddings-v2-base-en while setting new benchmarks for cross-modal retrieval, enabling seamless text-to-text and text-to-image search in one model. This makes it suitable for multimodal retrieval-augmented generation (MuRAG) applications.

Benchmark Performance

Text-Image Retrieval (Flickr30k and MSCOCO datasets):

Text-Text Retrieval on STS and BEIR benchmarks, jina-clip shows comparable or superior performance to jina-embeddings-v2-base-en, achieving a Spearman correlation of 0.8493 on STSBenchmark and NDCG@10 of 0.7161 on TRECCOVID.

The model is built with PyTorch, supports ONNX and Safetensors, and is released under the Apache-2.0 license. Its paper was presented at the MFM-EAI workshop at ICML 2024. For further details, see the full paper.