VLM2Vec LoRA

VLM2Vec is a multimodal embedding model that converts a vision-language model (Phi-3.5-V) into a unified embedding model capable of processing any combination of images and text to generate a fixed-dimensional vector based on task instructions. It is designed for massive multimodal embedding tasks including classification, visual question answering, multimodal retrieval, and visual grounding.

The model is trained on the MMEB benchmark (20 training datasets) using contrastive learning with in-batch negatives and GradCache to increase effective batch size. It is evaluated on 16 MMEB evaluation datasets covering both in-distribution and out-of-distribution tasks. VLM2Vec achieves an absolute average improvement of 10% to 20% over existing multimodal embedding models across all MMEB evaluation datasets.

The model processes any combination of images and text to generate a fixed-dimensional vector based on task instructions, using an [EOS] token as the representation of multimodal inputs. Unlike CLIP or BLIP, which encode text or images independently, VLM2Vec can follow task instructions to produce task-specific embeddings.

The VLM2Vec project has evolved through multiple versions. The V2.0 checkpoint (Apache-2.0 licensed) uses the Qwen2VL architecture. The MMEB benchmark has been extended to MMEB-V3, which covers 190 tasks including audio tasks (classification, cross-modal retrieval, temporal grounding), text retrieval (instruction-following, reasoning, long-context, multi-condition), and agent tasks (tool retrieval, GUI control, agent memory retrieval). MMEB-V3 also introduces OmniSET (Omni-modality Semantic Equivalence Tuples) for controlled analysis of modality effects and instruction-conditioned cross-modal retrieval behavior.