A user asks about the vacation policy and receives an answer based on a document that no longer exists. In this scenario, the Qdrant index still retains the document, and no one is logging which chunks are reaching the LLM. Measuring the quality of a RAG system begins at that layer: what it retrieves, in what order, and with what score.
Retrieval Decides the Outcome
The typical workflow converts the question into an embedding; Qdrant retrieves the $k$ nearest chunks using HNSW and sends them to the LLM. The bi-encoder encodes the question and the chunk separately, within the same model, and compares their vectors. This can elevate chunks that share the general topic, even if they do not contain the requested information.
Log the retrieved chunks, their scores, and the models used for every query. Without this log, a drop in quality doesn’t show what changed or at which point in the pipeline. The Hugging Face MTEB leaderboard gathers results by task and language; use it as a starting point and confirm the selection with your own queries. With that foundation, the next step is to re-rank the candidates before sending them to the LLM.
Re-ranking: A Second Filter with More Context
A cross-encoder receives the question and the chunk together and produces a relevance score for that pair. This approach evaluates the joint relationship, whereas cosine similarity only compares vectors generated separately. Running it over the entire corpus increases the cost; therefore, the common pattern consists of retrieving a short list with Qdrant, re-ranking it with the cross-encoder, and sending the highest-ranked chunks to the LLM. Options include the commercial Cohere Rerank API and the open-source BAAI/bge-reranker-v2-m3 model, available on Hugging Face.
Qdrant’s Query API supports prefetch and RRF fusion, allowing you to order results from both dense and sparse searches. Qdrant also supports multi-vectors for late interaction retrieval in patterns like ColBERT. By incorporating a re-ranker, both the order of the chunks and the context received by the LLM change. Repeat the response evaluation over a fixed set and compare the results before and after the change.
Retrieval Metrics That Actually Mean Something
To apply retrieval metrics, create a golden set with representative queries and their manually labeled relevant chunks. Recall@k measures what proportion of relevant chunks appear within the top $k$. MRR calculates the average reciprocal rank of the first relevant result. nDCG values the complete order and supports different levels of relevance.
Export queries from your logs and label relevance at the chunk level. A single document may be related to the topic but not contain the necessary answer. These metrics show if the system retrieves the evidence and places it in useful positions. To check if the final answer leverages that evidence, the generation must also be evaluated.
Evaluating Generation: Fidelity and Answer Relevance
Retrieving well does not guarantee answering well, because the LLM can ignore the context or add information that does not appear in the chunks. RAGAS can estimate the fidelity of the answer, its relevance to the question, and the precision of the context using an LLM as a judge. That judge is not equivalent to a ground truth source: a longer answer may receive a good score without providing more evidence. Complement automatic metrics with a manually reviewed sample on a periodic basis.
When the correct answer is reduced to a number, a name, or a deadline, an exact match or a deterministic rule is easier to audit than an LLM judge. If the RAG feeds agents with tools, the permissions, quotas, and logs of the MCP layer affect what the agent can do with the answer. The guide MCP in production: guardrails, quotas, and logs for your agents explains how to control that layer.
From the Golden Set to the CI Pipeline
Keep the versioned golden set alongside the code and run the evaluation with every change to the embedding model, prompt, or re-ranker. Establish which variations block deployment and which cases must be reviewed manually. When you change the embedding model, create a new collection and re-index the corpus; their vectors do not share the same reference space, even if they have identical dimensions. Testing both models in the same collection produces unreliable comparisons.
If data cannot leave your infrastructure, open-source embedding and re-ranking models can be run locally with appropriate infrastructure. Qdrant can also operate within your network, so the index does not depend on an external service. The guide Local AI and GDPR: privacy by design in your own infrastructure describes this approach.
Start with an initial set of real queries and label the relevance of each chunk. Calculate Recall@k and MRR before and after applying a cross-encoder to the first Qdrant candidates, using the same $k$ value for both runs. Run this evaluation in CI every time the model, prompt, or re-ranker changes. Every adjustment remains linked to concrete evidence: a relevant chunk that drops in position, a document that stops entering the context, or an unsupported answer.