Lodestone Base 4096 V1

lodestone-base-4096-v1 is an embedding model that maps long sentences and paragraphs to a 768-dimensional dense vector space for tasks such as semantic search, clustering, and similarity comparison.

Architecture and Capabilities

The model is based on a fine-tuned mosaic-bert-base-seqlen-2048 backbone and incorporates FlashAttention, Attention with Linear Biases (ALiBi), and Gated Linear Units (GLU). These architectural improvements allow the model to accept input sequences of up to 4096 tokens — eight times longer than most comparable sentence embedding models — while maintaining a manageable size suitable for both GPU and CPU inference. ALiBi enables extrapolation beyond the training length: the model was fine-tuned with a maximum sequence length of 2048 tokens but supports inference at 4096 tokens.

Training Data and Procedure

The model was fine-tuned on nearly 1.5 billion sentence pairs using a contrastive learning objective. The training data is a weighted sample from multiple public datasets, including Reddit comments (726M pairs), S2ORC citation pairs with abstracts (252M pairs), Reddit title-body pairs (127M pairs), Amazon reviews (87M pairs), WikiAnswers, PAQ, Stack Exchange, MS MARCO, and others. Full dataset details and weights are available in the original model repository.

Performance and Usage Notes

On standard text embedding evaluation benchmarks, lodestone-base-4096-v1 achieves performance comparable to widely used models such as all-mpnet-base-v2 while supporting a longer input context. The model outputs normalized embeddings via mean pooling over token representations. When used through the gigarouter API, no local installation or dependency management is required; the model is available as an OpenAI-compatible endpoint.