The bottleneck in enterprise AI isn't the model; it's what never reaches it. Here's how to design a retrieval layer that sends only the signal your LLM needs. That is indeed the context.
It isn't so much of a problem in structured data. The data is already engineered for it. But it is a complex aspect in the unstructured paradigm, especially when it is multi-modal and rich.
Why?
Every enterprise AI team hits the same wall. The demo worked beautifully on a thousand documents. Then someone plugged in the full corpus; two million PDFs, a decade of slide decks, scanned contracts, clinical imaging reports — and accuracy collapsed. The model didn't get worse. The context got worse.
This is the context problem, and it's the reason most RAG projects quietly fail between prototype and production. The fix isn't a better LLM. It's a better architecture for what reaches the model and what doesn't.
Is it unique to the industry?
Search teams spend engineering time around it in bleeding-edge tech like Spotify and Perplexity, because the right context is what makes the difference between a good product or not. Enterprises need to benefit from the same maturity, but investment in engineering for chunking isn't the focus yet — for various reasons, including budget.
The Context Budget Problem
Large language models have context windows measured in tokens. GPT-4o handles 128K. Claude handles 200K. That sounds like a lot until you do the math on an enterprise corpus. A single quarterly earnings deck might be 15,000 tokens once extracted. A regulatory filing runs 80,000. Multiply by the number of documents that are even remotely relevant to a given query, and you're choosing between two bad options: stuff the window and hope the model finds the answer, or truncate and lose the answer entirely. Stuffing the window isn't too great an idea either — the more context you feed a model, the less precise an answer you are going to get.
Finding the right context is an art in itself
What do companies do now?
So most companies end up not processing these documents. Or they do a single-document-based query. Or they outsource external document processing to licensed platforms. However, the intelligence often lies in an internal corpus that is not accessible to the external world; that's the treasure that needs to be mined.
In addition, most teams , even when they build a RAG , use option 1. They retrieve the top-k chunks by vector similarity, typically k=5 or k=10, concatenate them into the prompt, and send it to the model. It works on clean wiki pages. It fails on real enterprise data where the relevant information might be a chart caption on slide 14 of a 40-slide deck, buried in a paragraph that shares almost no lexical overlap with the query.
The result is token waste at scale. If your RAG pipeline sends 20,000 tokens per query and you run 10,000 queries a day, you're burning 200 million tokens daily; most of it noise. At current inference pricing, that's not a rounding error. It's a line item the CFO will notice.
Semantic Models vs. Semantic Search
There's a critical distinction that gets lost in most RAG tutorials: semantic search and semantic understanding are not the same thing. Semantic search finds documents or chunks that are similar to a query in embedding space. Semantic understanding extracts the specific meaning a query requires from within a document — regardless of surface similarity.
Consider the query:
"What was the patient outcome in the Phase II trial for compound XR-447?"
A semantic search engine might return chunks from any document mentioning "Phase II," "trial," or "XR-447" — including a press release, a competitor's filing, and a meeting note where someone mentioned the compound in passing. What you actually need is the outcomes table from the clinical study report, which might use entirely different language: "efficacy endpoint," "adverse event profile," "primary completion date."
A semantic model, in the architectural sense, understands these relationships. It knows that "patient outcome" maps to efficacy endpoints, that Phase II trials have specific document structures, and that the authoritative source for this information is the CSR, not the press release. This is what separates retrieval that works in production from retrieval that works in a Jupyter notebook.
Many companies use graph ontologies for it. It was a popular method until sometime ago. However, with the rise of better reasoning LLMs (most popular ones are good at this job) and search approaches like ranking and retrieval strategies, we can completely get away from this. And that in itself is the basis of this solution and blog: how do we build a truly functioning context layer for unstructured data without over-engineering on graph ontologies.
Anyone who has built a graph database in the past knows that building one is easy; maintaining it over time is not.
PS: This is not to say that this is not relevant in certain use cases (like disease synonyms, etc.), but it is not the need for all RAG origins.
Architecting the Context Layer
The architecture we recommend, and the one TheHaze implements, treats context as a first-class engineering problem with three distinct stages: ingestion, indexing, and retrieval. Each stage has its own optimization target, and conflating them is the most common architectural mistake we see.
Stage 1: Ingestion — Preserve Structure, Not Just Text
Most RAG pipelines treat ingestion as OCR-with-extra-steps. They extract plain text from PDFs, chunk it into 512-token segments, embed each chunk, and call it done. This destroys the structural information that makes enterprise documents useful: tables, charts, headers, footnotes, cross-references, and the relationship between visual and textual content on the same page.
Proper ingestion preserves document structure as metadata attached to each indexed unit. A chart on slide 14 of a board deck should be indexed as a chart — with its title, axis labels, data series, and the speaker notes on the same slide — not as a random paragraph extracted from the OCR output. This structural metadata becomes critical at retrieval time, when the system needs to decide whether to return a chart, a table, or a narrative paragraph.
Stage 2: Indexing — Multi-Representation, Not One-Size-Fits-All
A single embedding model applied uniformly across all content types is a compromise that satisfies no one. Text-heavy documents benefit from dense retrieval models. Structured data like tables and financial models benefit from hybrid search combining dense embeddings with sparse lexical matching. Visual content like charts, diagrams, and scanned forms needs multi-modal embeddings that capture layout and visual semantics, not just extracted text. The index should maintain multiple representations of the same content, each optimized for different query types. When a user asks a factual question ("What was Q3 revenue?"), the system routes to the representation most likely to contain a precise answer. When they ask an analytical question ("How did margins trend over the last four quarters?"), it routes differently. This query-aware routing is what keeps token budgets tight without sacrificing recall.
Stage 3: Retrieval — Rank, Don't Just Retrieve
Vector similarity is a starting point, not an endpoint. The retrieval stage should operate as a pipeline: broad semantic search to generate candidates, cross-encoder reranking to score true relevance, metadata filtering to enforce access policies and freshness requirements, and deduplication to avoid sending the same information twice in different formats.
The output of this pipeline isn't a list of document chunks. It's a structured context package: ranked, deduplicated, and formatted for direct injection into an LLM prompt. The package includes not just the content but provenance metadata: source document, page number, extraction confidence, and timestamp. This gives the model (and the user) the ability to verify answers rather than trust them blindly.
The Token Economics of Good Architecture
Let's quantify the difference. A naive RAG pipeline retrieving top-10 chunks at 512 tokens each sends 5,120 tokens of context per query. Accuracy on enterprise corpora typically lands around 60–70%, meaning three out of ten queries get wrong or incomplete answers, each one costing you the full inference run plus the downstream cost of human correction.
A well-architected context layer sends 1,500–2,500 tokens per query — less than half the volume — with accuracy in the 85–92% range on the same corpus. The token savings compound: lower input costs, faster inference, smaller context windows required (which opens up cheaper model tiers), and fewer failed queries requiring human intervention.
The goal isn't to fill the context window. It's to send the smallest amount of information that lets the model answer correctly. Every token you don't send is money saved and noise avoided.
Separating Knowledge from Generation
The final architectural principle — and the one that matters most for enterprise deployments — is keeping the knowledge base separate from the model. When retrieval and generation are fused into a single product, you inherit the vendor's indexing decisions, embedding choices, and context formatting. You can't swap models without re-indexing. You can't audit what context was sent to the model. You can't enforce data residency at the retrieval layer.
A separate context engine running in your VPC gives you independence. The knowledge base is a durable asset that outlives any single model generation cycle. The model is a commodity input — replaceable, upgradeable, and auditable. The retrieval layer is the moat: it's what makes your AI accurate on your data, not generic web-scale data.
What do we do?
This is the architecture TheHaze is built on. Not because it's theoretically elegant, but because we've watched too many teams burn six months and a seven-figure inference budget learning that the model was never the problem. The context was. Fix the context layer, and everything downstream gets better: cheaper, faster, and more accurate.
Where to Start
If you're evaluating or rebuilding your RAG stack, start with these three questions:
- What percentage of your corpus is unstructured or multi-modal? If it's above 40%, text-only chunking won't cut it. You need structure-preserving ingestion.
- How many tokens does your current pipeline send per query? If it's above 5,000, you're almost certainly sending noise. Measure accuracy at lower token budgets — you might be surprised.
- Can you swap your LLM without re-indexing? If not, your knowledge base is coupled to your model vendor. That's a strategic risk, not just a technical one.
If you have questions on designing an ideal RAG, we love the discussion — ping us at hello@thehaze.ai.
TL;DR
The teams that get this right treat retrieval as infrastructure: not a feature bolted onto a chatbot. They invest in the context layer first, measure accuracy on their own data (not benchmarks), and only then worry about which model sits on top. The model is the last mile. Remember, Context is everything.