An agent nobody measures is an agent that stops improving
You have imported your documentation, embedded the widget, and the agent is answering your users. Now what?
This is exactly where many companies stop — and where the agent stops getting better. A conversational agent is not a project you deliver once, it is a service you steer. The good news: four numbers are enough, they all sit in your SophiaDesk dashboard, and each one calls for a different action.
Metric 1 — Answer rate
What it measures: the share of questions your agent was able to answer from your documents.
This is your documentation coverage indicator. Beware of the common misreading: it does not tell you whether your answers are good, it tells you whether you have the material to answer at all. An agent that does not answer is not a broken agent, it is an agent missing a document.
SophiaDesk sorts this rate into explicit tiers:
| Answer rate | Reading |
|---|---|
| Below 20 % | Insufficient — the knowledge base needs content |
| 20 to 40 % | Useful, but clearly improvable |
| 40 to 60 % | Satisfactory — nearly one question in two finds an answer |
| 60 to 80 % | Very good — this is the target for a solid knowledge base |
| Above 80 % | Excellent |
The target to aim for: above 60 %. Below 50 %, the dashboard raises an explicit alert: at that point it is no longer fine-tuning, it is a structural gap.
What to do with it: a low rate is fixed with documents, not with technology. Identify the missing topics and import them.
Metric 2 — Unanswered questions
What it measures: the share of questions where the agent found nothing to answer with. Target: below 30 %.
It mirrors the previous metric, but it is far more actionable — because it is not just a percentage, it is a list. Every unanswered question automatically surfaces in your insights, with the exact wording the user typed.
Put differently: your users write your documentation roadmap themselves. They tell you, in their own words, what is missing.
What to do with it: work through that queue regularly. If a topic keeps coming back, it deserves a proper document rather than a one-off answer.
Metric 3 — Negative feedback
What it measures: the share of answers your users judged unsatisfactory. Target: below 10 %.
Do not conflate the two problems — they have neither the same cause nor the same severity:
- Unanswered — the agent found nothing, and says so. The user is disappointed, but not misled.
- Negative feedback — the agent answered, and the answer missed. This is more serious: either the documentation is ambiguous or out of date, or the question was misunderstood.
One red thumb is therefore worth several unanswered questions in terms of priority. Past 10 % negative feedback, SophiaDesk raises a health alert on the agent.
You can collect this signal two ways: classic thumbs feedback, or structured three-level feedback distinguishing “yes, perfect”, “not quite, let me rephrase” and “no, not at all”. The second is richer: it separates an incomplete answer from a wrong one.
What to do with it: every red thumb also surfaces as an insight. Read the question and the answer given, then fix the source document — not just the answer.
Metric 4 — Insights to complete
What it measures: the number of identified topics waiting for an answer from you.
This is the only one of the four you act on directly. The first three describe a situation; this one is a work queue. Each insight follows a simple cycle:
- To complete — the agent spotted a gap, you have not answered yet
- Active — you wrote the answer, the agent uses it immediately, alongside your documents
- Rejected — the topic is not relevant, it will not be suggested again
One piece of advice changes everything: be thorough in your insight answers. They do not only serve the user who asked, but everyone who will ask something similar.
What to do with it: an insight queue that keeps growing and never drains is the most reliable sign that an agent is being abandoned.
The routine: twenty minutes a week
Steering does not have to be heavy. A weekly slot is enough:
- Open the statistics over 7 days
- Check the answer rate, and above all its trend
- Work through the insights-to-complete queue — this is where the useful time goes
- Once a month, look at the word cloud of incoming questions: it surfaces topics you had not anticipated
Over a quarter, this routine does more for the quality of your agent than any technical setting.
Three reading traps
Judging on a single day. Daily volume is too small to mean anything. Read the 7-day smoothed trend, not today’s point.
Confusing volume with quality. A spike in questions is not a success in itself. A heavily used agent with a 30 % answer rate frustrates more users than it helps.
Forgetting the scope. Quality metrics cover billable requests: quick responses and requests blocked for security are excluded. That is deliberate — they would distort the reading.
Conclusion: measuring is already improving
These four numbers are not yet another dashboard. They form a loop: the answer rate tells you where you stand, unanswered questions and negative feedback tell you what to fix, insights give you the means to fix it.
The companies whose agents improve are not the ones with the best documentation on day one. They are the ones that look at these numbers every week.
No agent to steer yet? Create yours for free and import your first documents in minutes.