VLM2Vec Full

VLM2Vec-Full is a multimodal embedding model that converts any combination of images and text into a fixed-dimensional vector based on task instructions, built by fine-tuning the Phi-3.5-Vision-Instruct vision-language model (4.15B parameters, BF16 precision) on the MMEB benchmark. Unlike earlier models such as CLIP or BLIP that encode text or images independently without task instructions, VLM2Vec processes arbitrary image-text inputs to produce embeddings guided by a user-provided task instruction, enabling a wide range of downstream tasks including classification, visual question answering, multimodal retrieval, and visual grounding.

Key Strengths

• Trained with contrastive learning on MMEB-train (20 datasets) and evaluated on MMEB-eval (16 datasets covering both in-distribution and out-of-distribution tasks).
• Achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models on both in-distribution and out-of-distribution datasets in the MMEB benchmark, as reported in the VLM2Vec paper (arXiv:2410.05160).
• Supports four meta-tasks: classification, visual question answering, multimodal retrieval, and visual grounding, across 36 total datasets.
• Released under Apache 2.0 license.

Performance Overview

The model outperforms prior multimodal embedding baselines by a substantial margin across all evaluated tasks. Detailed results on the 36 MMEB evaluation datasets are available in the VLM2Vec GitHub repository and the associated paper.

Architecture

VLM2Vec-Full is a full-parameter fine-tuned variant of Phi-3.5-Vision-Instruct, using a contrastive training framework with in-batch negatives. It employs a batch size of 2048 during training and produces normalized embeddings via last-token pooling.