Why a language model alone cannot answer your questions

A language model does not know your company. It knows neither your price list nor your maintenance intervals nor the contract with customer Meyer. Ask anyway and you get a plausible-sounding answer — simply invented. Retrieval-augmented generation, or RAG, works the other way round: first your documents are searched, then the model formulates strictly on the basis of the passages found and names the source.

That shifts the decisive question from “which model” to “how well does the search find things”. And that is where most projects fail, not at the model.

Four architectures — and when each one carries

Vector RAG: documents are split into sections, embedded semantically and found by similarity of meaning. Strong for meaning-based questions in running text. Weak as soon as exact identifiers matter.
Hybrid RAG with reranking: semantic and classic keyword search run in parallel, the result lists are merged and a reranker re-sorts them against the original question. The standard case as soon as part numbers, standards or product names are involved.
Knowledge graph and GraphRAG: entities and their relationships are modelled explicitly. Necessary when answers span several hops — “which employee looked after the project of the customer whose contract contains clause X”.
SQL agent and SQL + RAG: metrics are computed, not estimated from similar-sounding text passages. For questions that need figures and explanatory documents at once, a router combines both paths.

Pure vector search reliably misses part numbers. If a wrong identifier is business-critical, there is no way around a lexical search path.

The invisible success factors

What decides quality is not visible in the finished interface. Chunking is about splitting documents along their subject-matter structure rather than by character count — a definition separated from its value becomes unfindable afterwards. Metadata carries source, version, area and permission along so that filtering can happen before every search. And the citation layer binds every statement to a concrete passage, including the ability to answer honestly with “I find nothing on that”.

That last point is the underrated one. A system that is not allowed to stay silent will guess.

Measure first, judge second

Retrieval and answer quality are measured separately. First the question: does the search surface the relevant passage among the top hits at all? Only then: does the model turn it into a correct answer? Conflating the two means optimising at the wrong end. This requires a gold set of typical, difficult, outdated and deliberately unanswerable questions — frozen before the build and signed off by the business.

How we support you

Data inventory and sample review before any architecture is fixed.
Architecture recommendation with reasoning — our AI readiness check gives you a first assessment in five minutes.
Building the pipeline from parsing and chunking through index and reranking to the citation layer.
Evaluation harness with a gold set and permission negative tests, so quality stays demonstrable rather than assumed.