How to tell if your chatbot is any good (evals, not vibes)

“It felt pretty good when we tried it” is not an evaluation. I hear it on intro calls more often than I hear a number. By then the widget is live, the model vendor has shipped two silent updates, and nobody can say whether the bot is closing tickets or inventing return windows.

If you cannot tell, you do not have a chatbot problem yet. You have a measurement problem.

Steal the questions from real life

Do not brainstorm “typical user queries” in a workshop. Pull them.

For a shop: the last two months of support tickets and live-chat logs. “Where is my order.” “Can I return opened headphones.” “Do you ship to Slovakia.” “This size chart does not match the photo.” Filter out the ones that are really “talk to a human about a damaged parcel”. Keep 30 to 50 that a bot is allowed to handle, and write the answer you would stand behind. The answer is the policy, not a vibe.

For an internal assistant: the Slack questions your team already asks the same two people. “What is our parental-leave policy.” “Where is the staging URL.” “Which SKUs are missing EAN.” Same treatment. One gold answer each, or a short list of acceptable phrasings.

This is tedious, and it is most of the value. A labelled set outlives the model, the prompt, and the vendor. When you swap the model next quarter, you rerun the same 40 questions. That is why you wrote them down.

Thirty is enough to start. Fifty is better. Two hundred can wait until you have production traffic to mine. Do not block launch on a perfect set. Do block launch on having none.

Score more than “it answered”

A single accuracy percentage hides the failures that cost money.

I score four things separately, on the same labelled set:

Correct. Did it match the gold answer, or a paraphrase a reviewer would accept? For “where is my order”, correct means the right status and a tracking path, not a polite paragraph.

Refusal. Did it refuse when it should? Medical advice, a legal interpretation of a contract, a discount it is not allowed to invent, a return exception that only a human can grant. A bot that never refuses looks helpful in a demo and expensive in chargebacks.

Groundedness. Did it stick to retrieved text, or did it add a fact? “14 days” when the policy says 14 days is fine. “14 days, and we’ll collect from your address for free” when that sentence is not in the policy is a fail, even if the rest is right.

Hand-off. When it should pass to a human, did it? A damaged parcel, an angry customer on the third contact, a SKU that is in the catalog twice. Deflection here is not a win.

A bot that is “95% correct” and calmly wrong on the other 5% is worse than one that says it does not know. Correctness without groundedness is how you get a support lead in your inbox asking why the widget promised next-day delivery to a village you do not serve.

You can start with a person scoring a spreadsheet. LLM-as-judge is fine later, on the easy dimensions, with a human spot-check. Do not begin there. You need to see the failures once with your own eyes or you will not believe the numbers.

Watch it after launch, or the set goes stale

Offline evals catch a prompt change that made refusals worse. They do not catch the week you add a new product line, or the week the carrier’s tracking page changes, or the week a model provider tweaks the default.

Log conversations (the privacy note on this site is the same rule I use with clients: collect what you need, say so, do not keep transcripts you will not read). Sample a slice every week. Promote the surprising ones into the labelled set. A question that appeared twenty times and is not in the set is a hole, not an anecdote.

This is the loop: labelled set, score, sample production, add the misses, rerun. If a vendor update drops groundedness, you see it on Tuesday, not after a month of tickets.

The numbers a shop actually cares about

“Accuracy” is a lab word. The operating numbers are simpler.

Deflection on the questions you wanted deflected. “Where is my order” should not become a ticket. Measure that against the same week last year, or against a holdout if you can.

Escalation on the questions you wanted escalated. A bot that “handles” a damaged-goods case by guessing is not saving money.

Hallucinated policy rate. How often a sampled chat invents a rule. This one should sit near zero. If it does not, stop adding channels and fix retrieval or the refusal policy.

If deflection falls and escalation rises, prompt tweaking will not save you. The catalog is incomplete, the policy PDF is a scan, or you asked the bot to cover a scope no document supports. That is a data and scope diagnosis. It is usually a week, not a rebuild.

I have seen teams spend a quarter on a new model and never look at these three. The widget still “felt fine”. The ticket queue did not.

If you already shipped something and cannot produce these numbers, start there. The eval set is the first deliverable, not a nice-to-have after the next prompt pass.