Unboxing Nomic Embed v1.5: Resizable Production Embeddings with Matryoshka Representation Learning

Unboxing Nomic Embed v1.5: Resizable Production Embeddings with Matryoshka Representation Learning

NEWSFebruary 14,2024

On February 1st, 2024, we released Nomic Embed - a truly open, auditable, and performant text embedding model. While the performance of an embedding model is often taken into consideration when evaluating it for production deployment, other factors including the memory, storage, and bandwidth requirements of the embeddings are also important to consider.

For example, storing the embeddings of a large dataset for a RAG app is often quite costly.

That's why we're excited to introduce Nomic Embed v1.5, which improves on Nomic Embed by giving developers the flexibility to explicitly trade off performance and embedding footprint.

Nomic Embed v1.5

We train nomic-embed-text-v1.5 with Matryoshka Representation Learning to enable variable embedding dimensions in a single model.

Nomic Embed v1.5 supports any embedding dimension between 64 and 768 as well as binary embeddings.

__wf_reserved_inherit
Training and Evaluation

We finetuned nomic-embed-text-unsupervised on our nomic-embed-text finetuning dataset. You can replicate the model and openly access the data in the nomic-ai/constrastors repository.

Nomic Embed v1.5 outperforms text-embedding-3-small at both 512 and 768 embedding dimensions.

At an embedding dimension of 512, we outperform text-embedding-ada-002 while achieving a 3x memory reduction. At a 12x memory reduction compared to nomic-embed-text-v1, our model performs similarly to all-MiniLM-L6-v2.

We found that our 768 dimensional performance is similar to nomic-embed-text-v1 while also enabling the variable degrees of freedom.

We did not evaluate text-embedding-3-small at lower dimensions because running the evaluation is prohibitively expensive and time-consuming as the model weights are not public.

You can explore the similarities and differences between different nomic-embed-text-v1.5 embedding dimensions using the custom mapping feature in the map below:

__wf_reserved_inherit
Matryoshka Representation Learning

Matryoshka Representation Learning is the technique that enables our model to have a variable embedding dimension.

__wf_reserved_inherit

Similar to the Matryoshka nesting dolls, we explicitly train our model to learn nested representations at different embedding dimensions. This allows us to truncate embeddings from the full size to reduce their memory footprint while retaining performance. For a more detailed explanation, please refer to the paper and this excellent blog post by Aniket Rege here.

Usage with Nomic Embedding API

Sign up to Nomic Atlas.

With the Nomic Embedding API, you can

curl https://api-atlas.nomic.ai/v1/embedding/text \
   -H "Authorization: Bearer $NOMIC_API_KEY" \
   -H "Content-Type: application/json" \
   -d '{ "model": "nomic-embed-text-v1.5",
         "texts": ["Nomic AI introduces Nomic Embed", "#keepAIOpen"],
         "task_type": "search_document",
         "dimensionality": 256}'

and in the official Nomic Python Client with

from nomic import embed
import numpy as np

output = embed.text(
   texts=[
       "Who is Laurens van der Maaten?",
       "What is dimensionality reduction?",
   ],
   model='nomic-embed-text-v1.5',
   task_type="search_document",
   dimensionality=256,
)print(output['usage'])

embeddings = np.array(output['embeddings'])

print(embeddings.shape)

# to get binary embeddings
embeddings = (embeddings > 0).astype(int)

Share this article:

Related Articles

Ask for it, and Nomic does it

The drawings, DWGs, schedules, and reviews Nomic does from your project files.

NomicPower layout
E-301 Power Plan Level 3.dwgDrawn by Nomic

AI CAD Drafting

Ask for the plan and get DWGs in your layers and text styles.

NomicDrawing review
Ashburn DC-3 100% CD Set.pdf2 findings

Automated Drawing Review

Coordination conflicts flagged on the sheet before issue.

NomicSchematic drawing set
M-101 HVAC Plan.dwgDrawn by Nomic

Schematic Drawing Sets

Architectural background in. Floor, plumbing, and HVAC sheets out.

NomicSubmittal review
08 44 13-01 Curtain Wall Shop Drawings.pdf3 findings

Automated Submittal Review

Shop drawings checked against the spec, cited to the page.

NomicReview comment response
M-501 Mechanical Details.dwgDrawn by Nomic

Review Comment Responses

Review letter in. A response and revised DWGs out.

NomicBIM takeoff
LEVEL 1LEVEL 2LEVEL 3LEVEL 4ROOF
HPMO-Structure.ifc

AI for BIM & IFC Models

Ask an IFC model for quantities and missing data.

NomicEnergy code schedules
Space and Lighting Schedules.xlsxDrawn by Nomic

Energy Code Schedules

Model spaces and lighting in. An energy-code import file out.

NomicCode compliance
A-201 Second Floor Plan.pdf1 finding

Code Compliance Review of Drawings

Drawings checked against 380+ codes, with citations.

NomicSurvey to DWG
M-001 Existing Conditions.dwgDrawn by Nomic

Existing Conditions Surveys

Site walk photos and notes in, a to-scale DWG and report out.

NomicDetail search
Lakeview Library CD Set.pdf

Firm-Wide Detail Search

Find the detail your firm already drew.

NomicRevit-ready DWG
P-201 Domestic Water Reroute.dwgDrawn by Nomic

Revit-Ready Deliverables

Coordination comments drawn as a DWG to link into your model.

NomicRFI response
G-003 Life Safety Plan.pdf

Project Research

Answers from every drawing, spec, and email on the job.

Nomic agents work in your project delivery software and tools
SharePoint
Egnyte
Autodesk Forma
Revit
AutoCAD
Procore
ProjectWise (Bentley)
Trimble ProjectSight
Fieldwire
Google Drive
Box
Dropbox
Microsoft Teams
Outlook
Gmail
Nomic Embed Matryoshka | Nomic News