RAG Embedding Models

RAG Embedding Models are the specialized models that create vector embeddings used to look up document chunks in the vector database. An appropriate embedding model is important for the accuracy of the RAG chunk retrieval, and using an efficient model is also important for latency of the overall RAG system. There are several different options for embedding models that can be used, with both commercial and open-source options

See also more research on related areas:

• Model evaluation
• RAG architectures
• Vector databases
• RAG system evaluation
• Advanced RAG architectures

Research on RAG Embedding Models

• Niklas Muennighoff, Nouamane Tazi, Loïc Magne, Nils Reimers, 19 Mar 2023 (v3), MTEB: Massive Text Embedding Benchmark, https://arxiv.org/abs/2210.07316 https://github.com/embeddings-benchmark/mteb
• HF, February 23, 2024, Introduction to Matryoshka Embedding Models, https://huggingface.co/blog/matryoshka
• Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, Ali Farhadi, 8 Feb 2024 (v4), Matryoshka Representation Learning, https://arxiv.org/abs/2205.13147 https://github.com/RAIVNLab/MRL
• OpenAI, January 25, 2024, New embedding models and API updates, https://openai.com/index/new-embedding-models-and-api-updates/
• Nomic Team, 2024, Introducing Nomic Embed: A Truly Open Embedding Model, https://www.nomic.ai/blog/posts/nomic-embed-text-v1
• Zach Nussbaum, John X. Morris, Brandon Duderstadt, Andriy Mulyar, 2 Feb 2024, Nomic Embed: Training a Reproducible Long Context Text Embedder, https://arxiv.org/abs/2402.01613
• Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, Han Xiao, 19 Sep 2024 (v3), jina-embeddings-v3: Multilingual Embeddings With Task LoRA, https://arxiv.org/abs/2409.10173
• John X. Morris, Alexander M. Rush, 8 Nov 2024 (v4), Contextual Document Embeddings, https://arxiv.org/abs/2410.02525
• Michael Wood, Nov 22, 2024, The Insanity of Relying on Vector Embeddings: Why RAG Fails, https://blog.cubed.run/the-insanity-of-relying-on-vector-embeddings-why-rag-fails-be73554490b2

More AI Research

Read more about:

• Inference Optimizations
• Loop Optimizations
• Code Optimizations
• « Research Home