LLM2Vec Llama 2 7B Chat (Supervised)

McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised is a text embedding model that converts the Llama-2-7b-chat decoder-only language model into a bidirectional text encoder using the LLM2Vec recipe, followed by supervised contrastive fine-tuning on publicly available data.

Method

The LLM2Vec approach (BehnamGhader et al., COLM 2024) transforms any decoder-only LLM into a universal text encoder through three steps: enabling bidirectional attention, training with masked next token prediction (MNTP), and unsupervised contrastive learning (SimCSE). The supervised variant adds a second LoRA adapter trained on public E5 data with supervised contrastive learning, building on the MNTP LoRA weights.

Performance

The unsupervised LLM2Vec recipe achieves a new state-of-the-art on the Massive Text Embeddings Benchmark (MTEB). When combined with supervised contrastive learning—as in this model—it reaches state-of-the-art on MTEB among models trained exclusively on publicly available data (as of May 24, 2024). The model uses mean pooling with a maximum sequence length of 512 tokens and is optimized for embedding tasks such as retrieval, semantic similarity, and classification.

Architecture and Usage

Built on Llama-2-7b-chat-hf with bidirectional attention enabled, the model uses LoRA adapters (MNTP + supervised) merged into the base model. It is available under the MIT license and is hosted on gigarouter as a managed API, requiring only a simple API call for inference.