Your AI Agent Says “Done!” — Here’s How to Know If It’s Lying

An AI agent can say “done” after a tool call failed. The dangerous part is not that the model makes a mistake; it is that the surrounding application may record a normal HTTP response, show a green status, and give the user a polished success message anyway. The same blind spot has a price when agents write the tests that are supposed to catch it — the hidden cost of letting AI agents write your tests.

Definition: A silent agent failure is a run where an underlying tool or step fails but the final answer does not acknowledge that failure.

Example: An order lookup returns an error, but a support agent tells the customer that the refund was processed.

Key takeaway: “No exception was raised” is a process signal, not proof that the agent did the right thing.

Business impact: If the system cannot connect tool outcomes to the final answer, incorrect actions can look exactly like successful ones.

The Towards AI article presents a practical answer: treat agent observability and agent evaluation as two separate disciplines. First, record the run as a structured trace. Then, evaluate the completed trace and answer against a rubric that can detect silent failure.

Why ordinary logs miss the lie

A traditional application often has a useful binary signal: the request either returned an error or it did not. Agents break that assumption because the model sits between a tool result and the user-facing answer.

A tool can fail with a timeout, a 404, a permission error or malformed data. The model may then continue, infer what probably happened, and produce a complete-looking response. If the wrapper catches the exception and returns a normal model message, the outer request can still finish with a 200 response.

A log entry such as agent_request completed tells the operator only that the orchestration process ended. It does not establish that the requested work was completed, that the answer reflects the tool result, or that the final claim is supported by evidence.

This is why a green request counter and a low exception rate are weak correctness metrics for agents. The failure is semantic: the system’s final claim is inconsistent with what happened underneath.

Observability and evaluation solve different problems

Observability records the run

Observability answers: What happened?

A useful agent trace records a hierarchy of spans. The root span represents the user request. Child spans represent the model call, retrieval step, tool call, validation step and final response. Each span can carry start and end times, status, error details, token usage, estimated cost and relevant metadata.

This is the same basic idea used in distributed systems. The OpenTelemetry traces specification describes traces as a way to represent the path of an operation through a system. For an agent, the path includes reasoning-related orchestration and external actions, not just network services.

A flat log might say that an order lookup happened. A trace can show that the lookup started, returned a 404, was followed by a model call, and ended with a success-shaped answer. That causal structure is the evidence needed for diagnosis.

Evaluation judges the result

Evaluation answers: Was the run good enough?

An evaluator reads the finished trace and final answer against explicit checks. Typical checks include:

• Did the agent complete the requested task?
• Did it use the appropriate tool?
• Did it acknowledge tool errors?
• Did it stay within latency and cost budgets?
• Did it return an answer supported by the available data?
• Did it escalate when the action required human judgment?

The important distinction is that evaluation does not merely check whether the code threw an exception. It evaluates the outcome and the relationship between the outcome and the recorded events.

A system can have excellent traces and still deliver bad answers if nobody scores them. It can also have a test suite that checks only crashes and still miss a confident false success. Production reliability requires both halves.

The silent-failure check

The most useful check in this pattern is deliberately simple: compare error spans with the final answer.

A basic version works like this:

1. Find tool and model spans whose status is error.
2. Preserve the error type and the affected operation.
3. Inspect the final answer for an explicit acknowledgement of the failure.
4. Fail the evaluation if an error exists but the answer claims successful completion without qualification.
5. Route ambiguous cases to human review rather than pretending the result is safe.

The check should not rely only on a blacklist of words such as “done” or “success.” It should consider the task, the tool result and the answer’s claim. “I could not verify the refund because the order service returned an error” is an appropriate response after a failed lookup. “Your refund has been processed” is not.

A keyword heuristic is a useful first guard, not a complete truth system. It can miss subtle claims, misunderstand a negation or flag a legitimate confirmation. The right architecture records the evidence, makes the rule explainable and adds stronger semantic or factual checks where the workflow justifies the cost.

A minimal production architecture

A dependency-light implementation can be organized into four parts:

The tracer should be created at the request boundary and passed through the agent. Tool wrappers should close spans even when exceptions occur. The final answer should be attached to the root trace so the evaluator can compare it with child events.

The evaluator should return separate checks rather than one unexplained score. A run might pass latency and cost while failing task correctness and silent-failure detection. That tells the engineering team what to fix and gives operations a reason for escalation.

Why the evaluator should not rerun the agent

A post-run evaluator should read the completed trace and final answer rather than rerun the original request. Rerunning can produce a different tool result, different model output or different side effects. It also makes it harder to know whether the evaluation describes the original run.

Keeping evaluation read-only has two benefits:

• Reproducibility: the same trace can be scored offline, in CI or during an incident review.
• Safety: a check cannot accidentally repeat a charge, send a message or mutate a record.

The same evaluator can therefore score recorded test scenarios and selected production traces without changing the agent’s behavior.

Testing the failure mode on purpose

The silent-failure case belongs in a regression suite. Create a deterministic scenario where the tool returns an error and the intentionally buggy agent produces a confident success message. The expected result is not a successful response; it is a failed evaluation with a clear explanation.

A useful test set includes at least these cases:

Tests should assert the reason for failure, not just a boolean. A future engineer needs to know whether a change broke tool selection, error propagation, answer grounding or the evaluator itself.

Production metrics that matter

Teams often start with request count, latency and exception rate. Keep those metrics, but add measures that reflect agent correctness:

• silent-failure rate;
• percentage of runs with a failed tool span;
• percentage of failed-tool runs acknowledged in the final answer;
• unsupported-success rate;
• evaluation pass rate by workflow and model version;
• escalation rate and human correction rate;
• cost and latency per successfully completed task;
• regression failures after prompt, tool or model changes.

The goal is not to score every sentence with an expensive judge. Use deterministic checks where the contract is clear, sample broader semantic evaluations where the risk justifies them, and keep the trace for investigation.

Common mistakes

Treating logs as traces

A collection of strings is not automatically a causal trace. Without parent-child relationships, timing and status, reconstructing a run becomes guesswork.

Treating uptime as correctness

A healthy process can produce an unhealthy answer. Infrastructure health and task correctness should be separate signals.

Using one opaque evaluation score

A single number hides whether the failure was cost, latency, tool choice, unsupported content or silent failure. Keep checks independent and explainable.

Evaluating only happy paths

The most valuable test is often the one where the tool is unavailable, data is incomplete, permissions change or two systems disagree. Agents need failure-path evaluation before they receive write access.

Letting the evaluator change the system

An evaluator should observe and score. It should not silently retry a payment, modify customer data or rerun an external action. Escalation must be explicit.

The practical rule

Before an AI agent is allowed to say “done,” the system should be able to answer four questions:

1. Which tools and data did the agent actually use?
2. Did any underlying step fail or return incomplete evidence?
3. Does the final answer accurately reflect those outcomes?
4. What independent check decided that the run was acceptable?

If the system cannot answer those questions, it does not know whether the agent succeeded. It only knows that the process stopped.

The fix is not necessarily a larger model or a more elaborate prompt. Instrument the run, preserve the evidence, evaluate the finished outcome and make silent failures visible before they reach a user.

FAQ

Is an agent that says “done” necessarily lying?

No. The phrase is a warning sign only when it conflicts with the recorded evidence. The correct response depends on the task, tool results, policies and final answer. The point of tracing and evaluation is to replace suspicion with evidence.

Can ordinary unit tests detect silent agent failures?

They can detect them when the test explicitly creates the failed-tool scenario and checks the final claim. Tests that assert only that no exception was raised will usually miss the problem.

Do I need a commercial observability platform?

No. A small structured tracer can record spans locally and support offline evaluation. Commercial platforms become useful when teams need retention, search, visualization, collaboration and integrations across many services.

Should every production run be evaluated by another LLM?

No. Deterministic checks should handle clear contracts such as error acknowledgement, required fields, tool status and budget limits. Use model-based judges selectively, with sampling, versioning and human review for high-impact decisions.