Agent Auditing Engine Tests More Than Final Answers

A²E is an end-to-end engine for auditing agent harnesses, not just scoring their final answers. The paper combines a shared task interface, automatically instrumented execution traces and lifecycle-aligned evaluation, then tests nine harnesses across 23 benchmarks. For teams comparing AI agents in production, the important shift is from asking “was the answer correct?” to asking how the system planned, used tools, recovered from errors and consumed resources; the full design is described in the authors’ paper on Hugging Face.

Definition: A²E (Agent Auditing Engine) is an evaluation engine that runs agent harnesses against shared tasks, records their execution and scores multiple dimensions of the resulting trajectory.

Example: The paper compares the same model and matched task settings across harnesses such as LangGraph, CrewAI, Agno, Google ADK and the OpenAI Agents SDK.

Key takeaway: Final-answer correctness can hide meaningful differences in planning, tool interaction, efficiency and recovery behaviour.

Business impact: A harness decision should be based on the cost and reliability of completed workflows, not only on whether a demo produces the right text.

Why final-answer correctness is not enough

Final-answer correctness is an incomplete signal for an agent harness because it can hide how the system reached the result. In A²E’s evaluation, the same backbone model and matched runtime settings are used across nine harnesses, yet the paper reports correctness ranges from 0.568 to 0.663 across the eight instrumented harnesses while mean token cost varies 3.5×, from 2,063 to 7,319 tokens. The practical takeaway is to pair outcome scores with execution metrics before selecting a harness for a real workflow.

Multi-turn tasks expose harness differences more clearly than simple single-turn questions. The paper says all nine harnesses receive identical scores on several single-turn tasks, while multi-turn tasks show wide spreads: τ-bench ranges from 0.00 to 0.60, GDPVal from 0.00 to 0.60, and traject-bench from 0.20 to 1.00. Operators evaluating an agent should therefore include tool-interactive and longer-horizon cases, because a harness that looks interchangeable on question answering may diverge under real execution.

How does A²E connect benchmarks to harnesses?

A²E uses the Agent Task Protocol (ATP) to separate benchmark adaptation from agent-harness execution. ATP is an internal software protocol with shared task objects for inputs, bindings, runners and traces; the paper’s Task Layer uses those objects to compose benchmark tasks with different harnesses without writing a separate integration for every pair. The concrete takeaway is architectural: teams can change the benchmark adapter or harness adapter independently, making matched comparisons easier to reproduce.

A²E’s Task Layer supports 23 benchmarks across coding, conversational, research and computer-use work. Each adapter turns source items into a common task input, while tool-use and sandbox tasks can add callable tools, initial state or a container environment without changing the boundary presented to the harness. The paper reports 1,035 scored runs from five sampled tasks per benchmark-harness pairing, so the useful comparison is a full matrix rather than one hand-picked demonstration.

What does the Monitor Layer record?

A²E’s Monitor Layer turns agent execution into standardized, hierarchical traces. It maps framework-specific model calls, tool executions, workflow transitions and asynchronous operations into semantic activities and OpenTelemetry-compatible spans with timing, status and parent-child relationships. For an operator, this is the evidence needed to distinguish a slow tool, an invalid action, a repeated model call or a failure that was hidden by a polished final response; agent observability and evaluation cover the same operational distinction in more detail.

The monitor captures intermediate execution rather than asking each agent to explain itself afterward. A²E records model calls, state transitions, errors, tool calls, latency, token consumption and generated artifacts during the run, then streams the records to a centralized server. The practical benefit is trace fidelity: an evaluation can inspect the events that actually occurred instead of relying on a model-generated summary of its own behaviour.

Which metrics does the Evaluation Layer add?

A²E organizes metrics by the agent lifecycle so a failure can be located as well as counted. Its taxonomy covers Reasoning, Action, Final Answer and Runtime Quality, with dimensions such as task understanding, planning, tool use, skill application, memory, answer correctness, task completion, efficiency and safety. The paper’s Evaluation Layer combines rule-based measures—including accuracy, task success, tool success, latency, token usage and cost—with LLM-based judgements for qualities such as reasoning, instruction following and safety; the takeaway is to keep operational and semantic checks in one comparable pipeline.

A²E stores trajectories and metric results in a database so new evaluations can reuse old runs. The paper describes structured records for tasks, runs, turns, model outputs, tool calls, errors, resource usage and metric results, with indexed queries across benchmarks, models and harnesses. That design allows a newly introduced metric or judge model to score an existing trajectory without rerunning an agent or repeating an external side effect, which is the same reason production teams use model monitoring tools rather than relying on one-time logs.

What did the harness comparison reveal?

A²E finds no universally dominant harness because the best configuration changes with the benchmark. In the paper’s matched experiments, the top-ranked harnesses differ across GDPVal, MMLU-Pro and τ³-bench, and the authors explicitly caution that the matrix is not a universal framework leaderboard. The operational takeaway is to test the model-harness pair on the workflow, failure modes and budget that matter to the business rather than copying a framework’s result from another task.

A²E’s case study shows why trajectory evidence changes the diagnosis. On the same τ³-bench task with the same GLM-5.2 API model, LangGraph reaches correctness 1.0 with 10,122 tokens, four LLM calls and three tool calls, while CrewAI receives correctness 0.0 after 96,704 tokens and five tool calls. The paper attributes the difference to execution behaviour: LangGraph moves from device checks to the account-level cause, while CrewAI keeps exploring device-level recovery actions; the takeaway is that a harness can change tool selection, context accumulation and recovery even when the model is held constant.

What should teams take from A²E?

A²E makes agent evaluation a system-level task rather than a model-only contest. Its 23-benchmark, nine-harness matrix, standardized traces and lifecycle metrics expose differences that a final-answer score can miss, while the authors’ results show that task and harness rankings move together. Teams deciding whether to deploy or replace an agent framework should therefore build a matched evaluation set, capture traces, measure cost and latency, and inspect failure paths before trusting an aggregate score.

The paper’s strongest practical message is that evaluation infrastructure must evolve with the agent stack. ATP lets new task and harness adapters share a boundary, the Monitor Layer preserves execution evidence, and the Evaluation Layer lets teams add metrics without rewriting the runner. That combination does not prove that any harness is safe or best for every deployment; it gives operators a repeatable way to find out where a specific model-harness combination succeeds, wastes resources or needs human review.