Judge an AI agent by correct completed tasks, answer accuracy, useful handoffs, errors, staff effort, customer outcome and total cost. Compare those with a defined baseline; conversation count alone is an activity metric.
Key takeaways
- Define eligible, complete, pending and failed before launch.
- Measure net staff time after review and correction.
- Include high-impact errors and human handoffs.
- Use real pilot data and label every assumption.
The problem and the first decision
Your new AI assistant answered 700 messages this month. That sounds impressive. But did customers get accurate answers? Did staff have less work to do? Were appointments actually confirmed, or only requested? How many issues had to be repaired afterwards? A useful AI agent should be evaluated by outcomes—not how busy it appears.
How do you measure whether an AI agent is working?
Define a narrow job, record how it is handled before automation, and compare the pilot using consistent definitions. Measure correctness, completed outcomes, human handoffs, errors, staff effort and operating cost. Avoid judging performance from messages sent, total conversations or response speed alone.
Define success before launch
For a dental reception pilot, the goal might be “collect complete appointment requests without false confirmations.” For a real estate enquiry assistant, the goal might be “deliver a usable buyer brief to an assigned agent.” For a home-service business, it might be “reduce calls made solely to collect missing job details.” Write down what counts as eligible, completed, pending and failed. An answer is not necessarily a completed task.
Seven useful metrics
1. Correct task completion rate — Among eligible requests, how many reached the agreed outcome correctly? A booking is complete only after the authorised system confirms it.
2. Answer accuracy — Were factual answers supported by the approved business source at the time they were given? Sample real conversations and review the source behind each answer.
3. Handoff usefulness — When AI escalated a case, did the employee receive enough context and a clear next action? Count cases where staff had to restart the conversation.
4. Error and unsafe-action rate — Track false confirmations, wrong recipients, unauthorised changes, duplicate records and other failures by severity. A low overall error rate can hide an unacceptable high-impact error.
5. Staff time — Measure how long employees spend on the same category of work before and after automation, including reviewing or correcting AI outputs.
6. Customer outcome — Depending on the job, measure whether the user received a clear response, a verified appointment, a successfully handled request, or appropriate human help.
7. Total cost — Include software usage, integrations, maintenance, testing, monitoring and employee review time. Do not compare the AI subscription alone with full employee salaries.
A simple pilot scorecard
Before automation, record a sample of 50 comparable requests if enough volume exists. After launch, review another appropriate sample over a defined period. For each request, log: intent, eligible yes/no, final outcome, correct yes/no, handoff needed, manual minutes, and any error. An illustrative example: Original workflow: 12 minutes of staff handling per eligible request. New workflow: 5 minutes of handling plus 2 minutes of review. Net staff time reduced: 5 minutes per eligible request.
Those numbers are hypothetical, not NORVOI or client performance results. A financial estimate can then multiply verified net minutes saved by an appropriate fully loaded cost of staff time, subtract ongoing operating costs, and account for implementation costs over an agreed period. For example, if the pilot saved time but created more customer complaints or incorrect records, calling it a success would be misleading.
Avoid four common measurement mistakes
Counting every assistant response as a resolved ticket. Treating a booking request as a confirmed booking. Ignoring staff time spent correcting inaccurate output. Comparing different periods with different workloads, categories or seasonality without noting the limitation. Also avoid selecting only successful conversation screenshots. Include failed, escalated and abandoned requests in the review.
Connect the metrics to the business
An enquiry assistant might deliver faster replies but still produce incomplete leads. An operations workflow might shorten processing time yet leave more duplicates. Ask: Did the improvement reach the next person in the process? For marketing-driven enquiries, distinguish traffic, form submissions, qualified requests, booked conversations and actual customers. Do not invent attribution when the data cannot support it.
Decide whether to expand, repair or stop
Agree on minimum safety standards and business outcomes before the pilot. Expand when accuracy and handoff quality meet the approved threshold and the business benefit is repeatable. Repair when the failure pattern is clear and correctable. Stop or roll back when high-impact errors or unreliable actions cannot be safely controlled. A useful pilot can teach you that simple rule-based automation was the better solution.
How NORVOI can help
NORVOI builds AI agents and automation around defined tasks, with testing and review. We can help choose meaningful baseline metrics and set up a small reporting process so the business can judge whether the system is worth keeping.
Frequently asked questions
Is the number of automated conversations a useful KPI?
It is an activity metric, not proof of value. Pair it with correct outcomes and staff effort.
What is a good AI agent accuracy rate?
There is no universal figure. Acceptable performance depends on the task and the consequences of mistakes; critical actions need stronger safeguards.
How do I estimate ROI without inventing savings?
Use measured net time savings and realistic operating costs over a defined period. Label assumptions clearly.
Should handoffs count as failures?
Not automatically. A correct escalation may be exactly what the agent was designed to do.
Make the pilot prove its value
The right question isn't “How many messages did the AI answer?” It's “Did the work get done correctly, and did the people involved benefit?” Bring the task your agent is meant to complete and a small sample of before-and-after handling times or cases. We can help define a scorecard that captures correct outcomes, human effort, errors and operating costs—without inventing ROI.
Have a similar task in your business? Start with the request, the information your team trusts and the point where a person should take over.