Why a language model alone cannot answer the company questions

A language model does not know the company. It knows neither the price list nor the maintenance intervals nor the contract with customer Meyer. Ask anyway and you get a plausible-sounding answer — simply invented. Retrieval-augmented generation, or RAG, works the other way round: first the company’s own documents are searched, then the model formulates strictly on the basis of the passages found and names the source.

That shifts the decisive question from “which model” to “how well does the search find things”. And that is where most projects fail, not at the model.

An open ring binder of mixed paper next to a screen showing only blurred blocks.

Four architectures — and when each one carries

Vector RAG: documents are split into sections, embedded semantically and found by similarity of meaning. Strong for meaning-based questions in running text. Weak as soon as exact identifiers matter.
Hybrid RAG with reranking: semantic and classic keyword search run in parallel, the result lists are merged and a reranker re-sorts them against the original question. The standard case as soon as part numbers, standards or product names are involved.
Knowledge graph and GraphRAG: entities and their relationships are modelled explicitly. Necessary when answers span several hops — “which employee looked after the project of the customer whose contract contains clause X”.
SQL agent and SQL + RAG: metrics are computed, not estimated from similar-sounding text passages. For questions that need figures and explanatory documents at once, a router combines both paths.

Pure vector search reliably misses part numbers. If a wrong identifier is business-critical, there is no way around a lexical search path.

The invisible success factors

What decides quality is not visible in the finished interface. Chunking is about splitting documents along their subject-matter structure rather than by character count — a definition separated from its value becomes unfindable afterwards. Metadata carries source, version, area and permission along so that filtering can happen before every search. And the citation layer binds every statement to a concrete passage, including the ability to answer honestly with “I find nothing on that”.

That last point is the underrated one. A system that is not allowed to stay silent will guess.

Measure first, judge second

Retrieval and answer quality are measured separately. First the question: does the search surface the relevant passage among the top hits at all? Only then: does the model turn it into a correct answer? Conflating the two means optimising at the wrong end. This requires a gold set of typical, difficult, outdated and deliberately unanswerable questions — frozen before the build and signed off by the business.

What this is not for

Dumping the filing system “as is” into an index.
Hoping a larger model will replace chunking, permission filters and measurement.
Answers without a source passage and without permission to stay silent.

What is in your hands afterwards

A data inventory plus a sample, before the architecture is fixed.
An architecture recommendation with reasoning — the readiness check remains the five-minute first cut.
The pipeline through to the citation layer; a gold set and harness, retrieval and answer quality measured separately.

Pale document slabs standing like architecture, a search beam hitting one slab and drawing a thin citation line.

How we support you

Data inventory and sample review before any architecture is fixed.
Architecture recommendation with reasoning — our AI readiness check gives you a first assessment in five minutes.
Building the pipeline from parsing and chunking through index and reranking to the citation layer.
Evaluation harness with a gold set and permission negative tests, so quality stays demonstrable rather than assumed.

Write to us AI readiness check All services