Why Does Your Chatbot Give Wrong Answers When Your Pages Are Correct?

Somebody forwards a screenshot. The assistant on your website has told a customer that returns are accepted within thirty days, and they are not, and the page saying so has been correct for two years. The first reaction in the room is always the same: the model is making things up, we should change the model.

Almost never. In our experience most chatbot wrong answers on a grounded assistant have nothing to do with the model’s reasoning. The assistant answered honestly from what it was handed. The failure happened earlier, in the part nobody looks at, where a question is turned into a search and a search returns some text.

There are four places that failure lives: retrieval, chunking, staleness and ambiguity in the source pages themselves. Work through them in that order and you will find the cause of most incidents in under an hour. Our parent company makes the same argument from the product side in why a chatbot gives wrong answers; this piece is the diagnostic procedure.

The bug is upstream of the model

A grounded assistant works in three moves. It converts the visitor’s question into a search, it retrieves a handful of passages from your content, and it writes an answer using those passages. Anything you did to ground the assistant in your own content affects the middle step only. The model does not know your returns policy. It knows the paragraph it was given, and it will write a confident sentence from whatever that paragraph says.

Which means a wrong answer is evidence about your retrieval and your content, not about the model’s intelligence. Swapping to a larger model when the retriever handed over the wrong paragraph produces a more articulate wrong answer. That is worse, because a fluent mistake is harder to catch.

So the working assumption we start from, every time: the assistant told the truth about what it read. Find what it read. Everything else follows from that one artefact.

Where chatbot wrong answers actually come from

Symptoms map onto causes fairly reliably. This is the table we run down when a transcript lands, before touching any configuration.

Retrieval: the assistant never saw the right page

The page is not in the index

Start with the least interesting possibility, because it is the most common. The page exists, a person can read it in a browser, and the indexer never got it. Content behind a login, inside a PDF, inside an image, inside a tab that only populates after a click, or on a page whose text is assembled entirely in the browser. That last one is why the choice between static, server-rendered or client-side stops being an architecture preference and becomes an answer-quality decision.

Test it the blunt way. Fetch the URL the way a machine would, without running any JavaScript, and search the response for the sentence you expect the assistant to quote. If the sentence is not there, no amount of prompt engineering will help, because the assistant was never in a position to know it.

The page uses your words, not theirs

The second retrieval failure is a vocabulary mismatch. A customer types "can I get my money back". Your page says "Refund Eligibility Criteria" and then talks about remediation of unsatisfactory purchases. Both describe the same thing. The retriever is matching meaning, and it will often bridge that gap, but it bridges it less reliably than most teams assume, especially for short questions with two or three words in them.

The fix is editorial rather than technical. Put the question in the page, in the phrasing people use, and answer it in the sentence directly underneath. This is the same discipline that makes documentation work as a growth channel, and it is one of the few content changes that improves search visibility and assistant accuracy with a single edit.

Chunking: found the page, cut it in the wrong place

Retrieval does not fetch pages. It fetches passages, because a whole page is too much text to hand a model for one question. So your content is split into chunks, and the assistant sees one chunk, not the article you wrote around it.

Now think about how policy pages are written. The rule goes in one paragraph and the exception goes three paragraphs later, under a heading called "Please note". A human reads both. A retriever returns the first, the model answers accurately from it, and the exception never enters the conversation. The answer is not invented. It is incomplete, which in a customer conversation amounts to the same thing.

The remedy is to write so that each rule survives being lifted out of its page. Repeat the subject in the paragraph instead of relying on "it". Keep the condition next to the rule it modifies. Avoid "as described above". A useful test is to read any single paragraph on its own and ask whether a stranger could act on it, which is exactly the standard a good FAQ answer already meets, and it is why well-written FAQ blocks tend to outperform long prose pages inside an assistant.

Write every paragraph as if it will be read alone by somebody in a hurry, because inside a retrieval system that is precisely what happens.

Staleness: two true pages from different years

This is the failure that produces the most embarrassing screenshots, and it is almost always self-inflicted. Somebody published a new pricing page and left the old one live because a campaign still linked to it. Somebody updated the shipping terms in the help centre but not in the blog post from three years ago that ranks better. The index now holds two answers, both of which were true once.

A retriever has no reliable way to prefer the newer statement unless you give it one. It is matching meaning, and the older page often matches better, because it was written when the topic was fresh and the copy was more direct. Recency is a property you have to supply: dates in the markup, a canonical page for each topic, and old pages retired rather than orphaned.

The operational habit that prevents this is unglamorous: whenever a fact changes, search your own site for the old fact before you close the ticket. Every place it appears is a place the assistant can find it. Teams running AI-enabled operations get this right by treating the knowledge base as a system of record with one owner per topic, rather than as a pile of pages anybody may add to.

Ambiguity: the page is correct and unclear

The last category is the uncomfortable one, because the page passes review. A person reads it, understands it, and signs it off. Then a machine reads it without any of the context that person had, and produces something defensible and wrong.

These are the patterns we look for first, and each one is a content edit rather than a configuration change.

• Cross-references. "As mentioned above" and "see the table below" carry no meaning once the passage has been separated from its page.
• Facts trapped in images or PDFs. A price list rendered as a picture is invisible. So is a specification table living only inside an attachment.
• Pages that narrate a change. "We have updated our delivery times" describes an event. State the current delivery time as a fact, then mention the change if it matters.
• Unnamed subjects. Three plans described on one page using "this plan" throughout gives a retriever nothing to tell them apart by.
• Hedged conditions. "Usually within a few days, depending on circumstances" is honest and unusable. Give the range, then the exceptions.
• Undated pages. With no publication or review date, nothing downstream can tell a current statement from an old one.

None of this is new writing advice. It is the same clarity that makes a page useful to a person who arrives from search with one question, which is why we treat assistant accuracy as part of content and organic growth rather than as a separate AI workstream. The assistant is simply a very literal reader who never skims and never asks a colleague.

The order to diagnose in

When a wrong answer is reported, resist the urge to open the prompt. Run this instead. Most incidents resolve at step three or four.

1. Get the exact question. Not the summary somebody typed into the ticket. The literal string the visitor sent, including the typo, because the typo may be the whole story.
2. Ask what was retrieved. Look at the passages the system fetched for that question. This one artefact resolves more incidents than every other step combined.
3. Read the retrieved passage as a stranger. If it genuinely supports the wrong answer, the content is the bug and you have finished diagnosing.
4. Check whether the correct page is indexed at all. Fetch it without JavaScript and search for the sentence. Absent means an indexing problem, not an answering problem.
5. Look for a competing page. Search your own site for the wrong fact. Old campaign pages and superseded help articles are the usual culprits.
6. Check the phrasing gap. Compare the words in the question with the words on the page. If they share almost nothing, rewrite the page around the question.
7. Only now consider the prompt or the model. If the right passage was retrieved and the answer still contradicts it, you have a genuine generation problem, and it is rarer than the other six put together.

Whoever owns this loop needs the same standing as any other reviewer, with a defined pass rather than a vague sense that somebody is keeping an eye on it. We set that out in reviewing AI-generated work before a client sees it, and the same structure applies to answers a machine gives your customers directly.

What should happen when there is no good answer

Here is the concession this argument needs. Some questions have no published answer anywhere, and no amount of retrieval work changes that. Negotiated pricing, an account-specific query, anything carrying a legal or medical consequence: these are not content problems and they should not be answered by an assistant at all.

So design the refusal properly, because the refusal is a product decision rather than a fallback. An assistant that says it does not know, names what it does cover, and puts the visitor in front of a person loses far less trust than one that guesses well. Silence is not the alternative to a wrong answer. A useful next step is.

That next step is where the widget earns its place. Flidu is built on that shape: website-informed answers alongside contact, information and conversion actions in one lightweight widget, so an unanswerable question becomes a handoff instead of a dead end. Our parent company covers the reasoning in when an answer is not enough, and the disclosure side sits in how a chatbot should tell visitors it is not a person.

If you take one operational habit from this, take the question log and a weekly reading of it. Ten minutes with last week’s unanswered questions tells you what your content is missing more accurately than any keyword tool, because those are real people asking in their own words. Then fix the page, not the prompt. When the underlying content needs rebuilding rather than editing, that becomes an AI development and integration conversation, and you are welcome to bring us the transcript that started it.