Every AI agent demo works. That is what a demo is for. The question that decides whether an agent programme survives its first budget review is narrower and much less exciting: can you show, with numbers, that it is doing the job? Most teams cannot — they can show a transcript, or a satisfied stakeholder, but not a measurement. This article is about fixing that before you need it.
The good news is that measurement is largely a design decision made at the start, not a reporting exercise bolted on at the end. Get it right and evaluation doubles as your safety net.
Refer & earn: 30% of a client’s first month, then 10% of every month after that, for up to 24 months per client. See the partner programme →
Why demos are a poor predictor
A demo runs the two or three cases the builder knows work. Production runs the distribution: the ambiguous request, the missing field, the policy that changed last week, the customer who writes in three languages. An agent’s reliability is a property of that distribution, so the only honest way to know it is to test against it.
There is also an architectural reason to measure early. Anthropic’s engineering guidance is explicit that agentic systems trade latency and cost for task performance — meaning an agent that succeeds more often may still be the wrong choice if it costs three times as much per task. You cannot make that trade-off without numbers.
Build the test set first
Before launch, assemble a set of real tasks with known correct answers — fifty is a workable start. Draw them from actual history rather than imagination: past tickets, past enquiries, past documents. Include the ugly ones deliberately: incomplete inputs, edge cases, the ten per cent that a human currently escalates.
Two rules make the set useful rather than decorative. First, the answers must be authored by someone who knows the job, not by the agent. Second, the set must be frozen and re-run after every model or prompt change, because agent behaviour is not stable across versions.
The metrics that actually matter
- Task success rate against the frozen test set, with the failures kept visible. One number, and it moves when anything changes.
- Escalation rate. How often the agent hands back to a human. Too low is dangerous — it means it is guessing. Too high means it is not yet earning its keep. The healthy band is defined by the risk of the task, not by a universal target.
- Containment. The share of incoming work resolved without human touch — but only meaningful when combined with success rate. Containment alone rewards confident mistakes.
- Cost per resolved task, all-in: model calls, tool calls and the human time still required. This is the number that compares an agent to a person.
- Latency, measured end to end. Agents iterate; iteration costs seconds as well as tokens.
- Grounding accuracy. Whether answers cite sources that actually support them. Sample and check this manually on a schedule — it is the failure mode that damages trust fastest.
Measure the human side too
An AI employee changes the work around it, and that effect is measurable. Track handling time for the cases it escalates, the rate at which its drafts are accepted without editing, and the time the owner spends per week supervising it. If supervision costs more than the task saved, you have learned something valuable early.
Also measure the boundary: how often does it attempt something out of scope? That number is your early warning for a scope problem, and it maps directly onto the “excessive agency” risk that OWASP’s Agentic Security Initiative highlights in its threat guidance.
What “good” looks like by function
- Customer-facing answers: high correctness, zero tolerance for invented facts, escalation kept cheap and fast.
- Document processing: extraction accuracy per field, not per document — errors cluster in particular fields.
- Drafting: edit distance between the agent’s draft and what a human actually sent. That gap is the real quality metric.
- Routing and triage: misroute rate, and the cost of a misroute when it happens.
Reporting it to the business
Keep it to a page: what the agent is for, what it handled, success rate against the frozen set, cost per resolved task, escalations, incidents, and one improvement shipped. A short, honest page that includes failures buys more credibility than a dashboard nobody reads — and it is precisely the evidence trail that governance frameworks ask for. NIST’s AI Risk Management Framework, voluntary and currently being revised under the White House AI Action Plan, is organised around exactly this habit of mapping, measuring, managing and governing continuously rather than in one heroic audit.
How Faida Workforce measures its AI employees. Faida Workforce builds a frozen test set of real tasks with every deployment, reports success rate, escalation rate and cost per resolved task from launch, and keeps a named human accountable for the numbers. We would rather tell you an agent is not yet ready than ship a demo that flatters itself. If you want the measurement plan before the build, start at faidaworkforce.com.
Five measurement mistakes to avoid
- Testing only the happy path. The distribution is the product.
- Letting the agent author its own expected answers. It will agree with itself.
- Reporting containment without success. Confident wrongness looks like productivity.
- Never re-running the set. A new model version is a new agent; re-verify.
- Measuring cost in tokens only. The number that matters includes human review time.
The organisations getting durable value from AI in 2026 are not the ones with the most impressive demonstrations. They are the ones who can answer, in a single paragraph, what their agent did this month, how often it was right, what it cost, and what they fixed. That paragraph is the whole game.
Sources
- Anthropic Engineering — “Building Effective AI Agents” (latency/cost trade-offs; simplest-solution guidance): anthropic.com/engineering/building-effective-agents
- OWASP Gen AI Security Project — “Agentic AI: Threats and Mitigations”: genai.owasp.org/resource/agentic-ai-threats-and-mitigations
- National Institute of Standards and Technology — AI Risk Management Framework (map, measure, manage, govern): nist.gov/itl/ai-risk-management-framework
- Microsoft Work Lab — Work Trend Index, annual report, 5 May 2026: microsoft.com/en-us/worklab/work-trend-index
- Stanford HAI — The 2025 AI Index Report: hai.stanford.edu/ai-index/2025-ai-index-report
- EU Artificial Intelligence Act — Implementation Timeline (last updated 31 August 2026): artificialintelligenceact.eu/implementation-timeline
Illustration: Faida Workforce. This article is editorial content published under our editorial policy, including our disclosure on the use of AI in production.
30% of their first month, then 10% every month after that
Refer a business to Faida Workforce and you earn 30% of the client’s first paid month, then 10% of every month for up to 24 months per client — rising to 15% from the accrual after you pass 10 active clients. Paid by PayPal, batched on the 1st.
10 Pro clients = $2,598.70 over 24 months — $3,748.13 at the 15% tier.
Figures are potential earnings worked from the published rates, assuming every referred client pays for the whole cap — not a promise, not an average. The $1 trial is never commissionable.
