Your chatbot has your documents. Does it retrieve the right answer?

Retrospective for 3–9 August 2026.

You have uploaded your PDFs, catalogue and procedures to an assistant. Yet some answers remain vague or miss an important exception. Adding more documents does not always solve the problem. You also need to examine what the assistant retrieves when it prepares an answer.

A guide published by n8n on 6 August 2026 explores the trade-offs of semantic chunking in retrieval-augmented generation, or RAG. This approach groups passages by meaning rather than splitting them solely at a fixed length. It is not universally better: the right choice needs testing against the documents in question.

The problem may sit between two paragraphs

A document assistant does not necessarily reread the entire library for every question. Part of the system retrieves passages considered relevant, then the model prepares an answer using those extracts. If a condition and its exception are separated, an answer can be incomplete even when both facts exist in the files.

This changes how a knowledge base should be prepared. Asking a model to “answer accurately” is not enough. Check that it receives the right material and knows how to identify information it cannot find.

Example: delivery with an exception

Consider an illustrative scenario. A document states that delivery is available in a particular area. A later paragraph explains that certain bulky items require prior confirmation. If the assistant retrieves only the first passage, it may answer too confidently when a customer asks about an item covered by the exception.

The useful test is therefore not just “where do you deliver?” Ask whether this particular product can be delivered to that destination without an additional check. Test the real decision that the answer could trigger, rather than a general question alone.

Start with questions, not files

List the questions your team receives frequently. Add ambiguous requests, cases where documents conflict and questions the assistant should not answer. For each, record the expected answer, its source and any qualifications that must remain in the response.

This small test set lets you compare configurations against the same business need. It prevents one successful demonstration from becoming the entire evaluation. The valuable outcome is not an assistant that answers everything. It is one that responds usefully when it has evidence and stops appropriately when it does not.

Make the documents maintainable

An old catalogue and a current catalogue should not be treated as equally authoritative. Each source needs an owner, a scope and an identifiable version. Duplicates and obsolete text make verification harder, even with a capable model.

Also separate material suitable for customer-facing answers from internal team documents. Access controls remain necessary after indexing. A shared knowledge system must not make internal information available to every user of the assistant.

Show why an answer can be trusted

Where appropriate, connect the answer to its reference document. The team can then verify the information and fix the source instead of adding contradictory instructions to the prompt. Run the tests again when an important document changes.

Smart Wise Monkey’s AI consulting can help structure this work before connecting the material to a conversational assistant. A well-prepared knowledge base is a working business resource, not simply a folder of files handed to a model.

Let’s test your knowledge base against the questions your customers actually ask.