Choosing an embedding model for enterprise retrieval has become an exercise in leaderboard tourism. A team picks the top model on MTEB, plugs it into LangChain, runs a demo on ten documents, and declares victory. Three months later, accuracy in production is 58% and nobody can explain why. The model hasn't changed. The data has — because production data was always going to be different from demo data, and public benchmarks were never designed to predict performance on your corpus.
This article is a practical framework for benchmarking retrieval on your own data. Not theory. Not leaderboard scores. A repeatable process that tells you which embedding model, ranking pipeline, and chunking strategy actually works before you commit to a production deployment.
Why Public Benchmarks Mislead
The state of the art in embedding models moves fast. Models like Nomic Embed, Voyage AI, Cohere Embed v3, and open-weight alternatives from the E5 and BGE families regularly top public benchmarks. MTEB (Massive Text Embedding Benchmark) has become the de facto reference, covering dozens of tasks across retrieval, classification, and clustering.
The problem is that MTEB tasks are predominantly clean, text-heavy datasets: Wikipedia passages, scientific abstracts, news articles. Enterprise corpora are none of these things. They're scanned PDFs with OCR artifacts, PowerPoint decks where the critical information is in a chart rather than the bullet points, legal contracts with cross-references that span fifty pages, and Excel models where the answer lives in a cell formula, not adjacent text.
A model that scores 62.5 on MTEB retrieval might score 41 on your internal benchmark — while a model that scored 59 on MTEB scores 78. We've seen this repeatedly. The gap between public benchmark performance and enterprise performance is not a rounding error. It's the difference between a system that works and one that doesn't.
Building Your Evaluation Set
Before you test a single model, you need a ground-truth evaluation set built from your actual corpus. This is the step most teams skip, and it's the step that matters most.
Step 1: Sample Your Corpus Strategically
Don't random-sample. Stratify by document type, age, and format. If your corpus is 60% PDFs, 25% slide decks, and 15% spreadsheets, your evaluation set should reflect that ratio. Include edge cases deliberately: scanned documents, documents with complex tables, multi-language content, and documents that cross-reference each other.
A minimum viable evaluation set is 100 query-document pairs. A robust set is 500-1,000. Each pair consists of a natural-language query (the kind a real user would ask) and the specific document or passage that contains the correct answer. The query should be written by someone who knows the corpus — a domain expert, not an engineer guessing at user intent.
Step 2: Define Success Precisely
"Did it return the right document?" is necessary but not sufficient. Define success at multiple levels:
- Recall@k: Is the correct document in the top k results? (Typically k=5, k=10, k=20)
- MRR (Mean Reciprocal Rank): How high is the correct document ranked? First result is dramatically better than fifth.
- Passage accuracy: Did the system return the specific passage containing the answer, not just the right document?
- End-to-end accuracy: Given the retrieved context, does the LLM produce the correct answer? This is the metric that actually matters to users.
Track all four. A system with high Recall@10 but low MRR is finding the answer somewhere in the results but burying it — which means you're wasting tokens sending nine irrelevant chunks alongside the one good one.
Testing Embedding Models
With your evaluation set ready, test embedding models in isolation before adding ranking complexity. This tells you which models have the best raw retrieval signal on your data.
What to Test
Run at minimum these model categories against your evaluation set:
- Open-weight generalists: BGE-large, E5-mistral, Nomic Embed — strong baseline, no API cost, runs in your VPC
- Commercial API models: Voyage, Cohere Embed, OpenAI text-embedding-3 — often stronger but with data residency implications
- Multi-modal models: If your corpus includes charts, images, or scanned documents, test models that embed visual content natively (ColPali, Nomic Embed Vision, or similar)
- Domain-fine-tuned: If you have enough labeled data (1,000+ pairs), fine-tuning an open-weight model on your corpus often beats any off-the-shelf commercial model
For each model, index your full evaluation corpus using identical chunking parameters. Run all queries. Record Recall@k, MRR, and passage accuracy. Do not tune chunking per model yet — you want to isolate embedding quality first.
What We've Learned
Across dozens of enterprise benchmarks we've run, a few patterns hold:
Commercial models (Voyage, Cohere) consistently outperform open-weight models on clean text corpora — but the gap narrows or reverses on noisy, OCR'd, or multi-modal content. Multi-modal embedding models are not optional if more than 20% of your corpus contains charts, diagrams, or scanned pages. Fine-tuning on 2,000+ domain-specific query-passage pairs beats every off-the-shelf model we've tested, but requires an upfront investment most teams haven't made yet.
The single biggest variable isn't the embedding model — it's the chunking strategy. A mediocre model with structure-aware chunking beats a SOTA model with naive 512-token splits on enterprise data every time.
Testing the Full Pipeline
Embedding model selection is step one. Production retrieval is a pipeline, and the pipeline components interact in ways that benchmarks-on-isolation can't capture.
Chunking Strategy
Test at least three chunking approaches on your best-performing embedding model:
- Fixed-size chunking (256, 512, 1024 tokens) — the default, rarely the best
- Structure-aware chunking — split on headings, paragraphs, table boundaries, and slide transitions
- Semantic chunking — use embedding similarity to detect topic boundaries within documents
Measure not just retrieval accuracy but token efficiency: how many tokens does the pipeline send to achieve a given accuracy level? A chunking strategy that achieves 85% accuracy at 2,000 tokens beats one that achieves 87% at 8,000 tokens ; the 2% accuracy gain isn't worth 4× the inference cost.
Reranking
Add a cross-encoder reranker (Cohere Rerank, BGE-reranker, or similar) on top of your best embedding + chunking combination. Reranking is cheap relative to LLM inference and typically lifts MRR by 15-25%. Test with and without to quantify the lift on your data ; it's not always worth the added latency.
Hybrid Search
For corpora with significant structured content (legal, financial, technical), test hybrid search combining dense embeddings with BM25 sparse retrieval. Dense search finds semantically related content; sparse search finds exact matches for names, codes, and identifiers. The fusion of both consistently outperforms either alone on enterprise data , but the optimal weighting varies by corpus and must be tuned on your evaluation set.
Measuring What Users Actually Experience
Retrieval metrics are necessary but not sufficient. The ultimate benchmark is end-to-end: given a user query, does the system produce a correct, verifiable answer?
Run your top two pipeline configurations through a full RAG loop: retrieve context, inject into an LLM prompt, generate an answer. Compare the generated answers against ground truth using both automated metrics (exact match, token-level F1) and human evaluation on a sample of 50-100 queries. Human evaluation catches failure modes that automated metrics miss ; correct-sounding answers that cite the wrong source, partially correct answers that omit critical caveats, and confident hallucinations grounded in irrelevant retrieved context.
A retrieval pipeline that scores 90% on Recall@5 but produces wrong LLM answers 30% of the time has a context formatting problem, not a retrieval problem. Benchmark the full loop, not just the retrieval step.
Operational Benchmarks
Accuracy isn't the only dimension. Before choosing a production configuration, measure:
- Indexing throughput: How long to index your full corpus? Can you re-index incrementally?
- Query latency: p50 and p99 retrieval latency at expected query volume. Sub-200ms p50 is the target for interactive use.
- Index size: Memory and storage requirements at full corpus scale. Multi-modal indexes can be 3-5× larger than text-only.
- Incremental update cost: What happens when 10% of the corpus changes? Full re-index vs. incremental update makes a huge operational difference at scale.
Putting It Together
Benchmarking retrieval properly takes one to two weeks for a first pass , not because the tests are slow, but because building a good evaluation set requires domain expertise and careful curation. That investment pays for itself the first time it prevents you from deploying a pipeline that looks great on MTEB and fails on your contracts archive.
The process, summarized:
- Build a stratified evaluation set of 100-500 query-document pairs from your actual corpus
- Test embedding models in isolation >> measure Recall@k, MRR, and passage accuracy
- Test chunking strategies on your best model >> optimize for accuracy per token, not accuracy alone
- Add reranking and hybrid search >> measure the lift on your data, not someone else's
- Run end-to-end RAG evaluation with human review on a representative sample
- Measure operational metrics at full corpus scale before committing to production
Public benchmarks are useful for narrowing the candidate set. They're not useful for making the final decision. That decision belongs to your data, your queries, and your users. The teams that benchmark properly deploy once and scale confidently. The teams that skip it deploy three times and learn the same lesson each time — just with a different model.
Want the full playbook? Download our extended guide on designing retrieval benchmarks for your own questions and corpus.