AI Evaluation Needs Human–AI Team Results

AI evaluation is usually framed as a contest between a model and a human. A position paper accepted to the ICML 2026 Position Paper Track argues that this framing misses a more useful question: how effectively can people and AI achieve a shared goal together?

Definition: Human–AI team evaluation measures the performance of people and AI as one collaborative system, rather than treating the AI as an isolated competitor.

Example: A software benchmark could compare a developer alone, an AI alone, and synchronous or asynchronous human–AI teams on the same real issue.

Key takeaway: A model that wins a solo leaderboard is not automatically the best teammate.

Business impact: Companies choosing AI for real workflows should measure completed outcomes, human effort, trust and control alongside standalone model capability.

The authors—Jan Kulveit, Gavin Leech, Tomáš Gavenčiak and Raymond Douglas—present the argument in a paper published on arXiv. Their claim is not that autonomous benchmarks are useless. It is that treating autonomous performance as the main definition of progress can steer AI toward replacing human effort while under-measuring the capabilities needed for effective collaboration.

What is wrong with the replacement paradigm?

The paper calls the dominant approach a replacement paradigm: an AI system is tested against a human baseline, and progress is celebrated when the system performs a task autonomously or exceeds the human score. That design is useful for screening certain capabilities, but it also makes substitution look like the default destination. For AI teams, the practical takeaway is to keep solo capability tests while adding a separate measure of whether AI improves the person doing the work.

The authors argue that replacement-focused benchmarks also narrow the definition of intelligence. They rarely reward an AI for asking a useful question, expressing uncertainty, adapting to a user’s expertise, explaining its reasoning in a usable way or helping a person learn. For a human–AI workflow, those behaviors can determine whether the system is productive or merely impressive in isolation, so evaluation should record them explicitly.

The paper also points to interaction effects that solo tests cannot reveal. It cites prior work showing that maximizing standalone accuracy can reduce overall human–AI team utility when people struggle to predict or calibrate to a complex model. For an AI agent deployed with a human reviewer, the lesson is concrete: measure the quality of the combined decision and the review burden, not just the model’s answer on a fixed dataset.

How would human–AI team evaluation work?

Human–AI team evaluation treats one or more people and one or more AI systems as a combined unit working toward a common objective. The core test is whether the team achieves more than either participant could achieve alone, or achieves the same goal with better quality, speed, insight or resilience. For companies, that means defining the workflow outcome before choosing a model and comparing human-only, AI-only and human–AI conditions where the task allows it.

The paper proposes three groups of metrics rather than one universal score:

The paper's Table 1 makes the dimensions intentionally broader than accuracy: a team can finish a task quickly and still fail if the human does not understand the recommendation, cannot detect an error or feels unable to override the system. A useful evaluation therefore reports the task result and the conditions under which the result was produced, rather than collapsing collaboration into one speed or accuracy score.

Which AI capabilities do current benchmarks undervalue?

The proposed shift would reward AI systems that improve the human partner’s judgment rather than only producing a stronger standalone answer. The paper highlights useful questioning, scaffolding human learning, stimulating curiosity, communicating uncertainty and supporting mutual explanation. Product teams can turn those ideas into tests: did the AI ask for missing context, make uncertainty visible, adapt to the user’s goal and leave the person able to review the decision?

That emphasis matters for AI agents because agents operate across multi-step workflows rather than one isolated response. The paper's discussion of uncertainty, clarification and adaptability shows why an agent that completes a task but hides uncertainty may score well on output completion while increasing operational risk. An agent that knows when to ask, explain, pause or escalate may create more value even when it does less work autonomously.

Should standalone benchmarks be abandoned?

No. The authors explicitly argue for a layered evaluation programme, not a replacement of one benchmark family with another. Standalone tests remain useful for fast, cheap and scalable capability screening, while collaboration surrogates can support rapid iteration and real-participant studies can provide deployment-critical evidence. For a business, that suggests a staged budget: screen models alone, test the interaction design with a manageable proxy, then involve real users before expanding a high-impact workflow.

The paper uses autonomous driving to make the distinction clear. The driving system needs robust standalone testing, but deployment is still a human–AI system involving drivers, fleet operators, road users and regulators. The practical implication is that safety-critical AI needs both independent capability evidence and evaluation of how the complete system behaves when people monitor, intervene and take over.

Why is human participation difficult to scale?

Human–AI evaluation costs more than a static benchmark because researchers must recruit participants, coordinate interactive tasks and account for differences in skill, motivation and strategy. Subjective measures such as satisfaction, cognitive load, shared awareness and trust also require careful study design. Those constraints explain why the field has preferred automated leaderboards, but they do not show that solo scores are sufficient for deployment decisions.

The paper suggests shared interactive infrastructure, standardized environments and phased evaluation to lower the cost. It also discusses AI-based human surrogates that could provide faster, less noisy iteration. The authors caution that surrogates may underrepresent cognitive bias, fatigue, emotion and the diversity of real users, so surrogate results should accelerate screening rather than stand in for real participants when the decision is consequential.

Human variability can also become a useful test condition. The authors identify the user’s skill at working with AI as a confounding factor and a research priority. A robust evaluation should therefore report who participated, how experienced they were and whether the AI adapted to different users instead of hiding variation behind one average score. Background: Why AI writing detectors weaken trust.

What could a first team benchmark measure?

The paper proposes software engineering as a practical starting point because real GitHub issues provide observable outcomes and interaction logs arise naturally. A minimal study could compare four conditions: human alone, AI alone, synchronous human–AI collaboration and asynchronous human–AI collaboration. The same issue set, repository context and acceptance criteria would apply across conditions.

Initial measures could include issues resolved, code quality assessed through tests and review acceptance, time to resolution, the frequency and correctness of accepting or overriding AI suggestions, and unassisted performance before and after the AI-assisted session. For engineering leaders, the takeaway is to test whether the AI improves the developer’s result and judgment—not just whether the model can produce code without a person.

What should teams take from the paper?

The paper is a position paper and a research agenda, not a universal production scorecard. Its contribution is to make the evaluation target explicit: standalone benchmarks answer whether an AI can perform a task alone, while team benchmarks answer whether people and AI work better together. Companies should not collapse those questions into one leaderboard number.

The paper's proposed software-engineering benchmark provides a concrete starting pattern: choose one workflow with a clear definition of success and compare human-only, AI-assisted and, where meaningful, AI-only performance. Track outcome quality, time, corrections, escalation, human understanding, trust calibration and the ability to recover from errors. That connects the paper’s research proposal to the same operational question behind measuring AI automation ROI: did the system produce better verified work, and what did the people around it have to do?

The broader argument is about incentives. If evaluation rewards only autonomous replacement, development will tend to optimize for systems that work alone. If evaluation also rewards agency, learning, calibrated trust and collaborative performance, the field has a stronger reason to build AI that helps people do more—and helps them remain able to judge the result.

FAQ

Is human–AI team evaluation a replacement for model benchmarks?

No. The paper argues that standalone benchmarks and team evaluations answer different questions. Standalone tests are useful for screening basic capabilities quickly and at scale. Team evaluations measure the outcome and interaction of people working with AI. A deployment decision may need both: the first to establish what the model can do alone, and the second to establish whether the complete human–AI workflow is effective, safe and understandable.

What does “trust calibration” mean in an AI team?

Trust calibration means that a person relies on AI when it is likely to be right and questions or overrides it when it is likely to be wrong. A team can have high trust but poor calibration if users accept confident errors. A useful evaluation therefore measures not only whether users like the AI, but whether their acceptance and override decisions track the system’s actual reliability and uncertainty.

Why include learning and agency in an AI benchmark?

Learning and agency describe whether the human remains capable of understanding and directing the work. An AI system can increase short-term output while making the user passive, confused or unable to challenge a bad recommendation. The paper treats skill enhancement, cognitive load and agency as human-centric outcomes because a strong human–AI team should improve the result without removing the person’s meaningful control.

How should a company start evaluating an AI workflow?

Start with one workflow that has a clear definition of success, a measurable human baseline and manageable risk. Compare human-only work with AI-assisted work using the same inputs and acceptance criteria. Record outcome quality, time, corrections, escalations and the human’s ability to detect and override errors. Use standalone model tests as capability evidence, but do not treat them as proof that the full workflow will improve.