Loop Engineering: Better Feedback Is Not Enough

A loop-engineering experiment published by Towards AI on July 27, 2026 found that two ideas often presented as the foundation of reliable agent workflows are easy to connect and easy to mismeasure. The author built a feedback loop and a maker-checker verifier, then found that the feedback signal was initially empty and the “safe” verifier still accepted wrong code. The practical lesson is to validate the evaluator before trusting the loop.

Definition: Loop engineering is the design of a repeated agent process that generates work, observes evidence, decides whether to continue, and stops at a bounded condition.

Experiment: The author implemented a run-until-done loop and a maker-checker verifier in about 600 lines of Python, connected them to claude-opus-4-8, and evaluated them against MBPP+ tasks.

Key takeaway: Real feedback improves an agent loop only when the feedback contains a failure the agent can act on.

Operator impact: A verifier can reduce false accepts without proving correctness, so high-impact workflows still need an independent source of truth and a human gate.

What did the experiment test?

The experiment tested two loop-engineering components for AI coding agents: a run-until-done loop that feeds test failures back into the next attempt, and a maker-checker design that prevents the code-writing model from being the only judge of correctness. The author reports building both from scratch in roughly 600 lines of Python, using claude-opus-4-8, and evaluating them against MBPP+, a benchmark of small Python programming tasks. A related explanation of fresh-context verification shows why separating the authoring context from the review context can expose different assumptions. For teams assessing agent loops, the useful comparison is not “agent versus no agent”; it is whether each loop component adds trustworthy evidence.

The feedback loop generated a candidate solution, graded it, and retried failed problems with the actual error instead of a generic instruction to try again. The control arm used the same process but replaced the error with generic feedback. Because the two arms differed only in the information returned after failure, the comparison isolated whether the feedback channel carried useful signal. The broader harness, loop, and graph engineering distinction is useful context for separating the model from the control system around it. That design gives operators a simple test: compare actionable feedback with “try again” before claiming that a loop improves performance.

Why did real feedback fail first?

The first feedback run solved 32 of 35 problems in all three conditions: single-shot generation, real feedback, and generic feedback. The result did not show that feedback was useless; it showed that the hidden-test harness returned only a bare AssertionError, with no failing input, expected value, or actual value. Since the model could not see what to fix, “real feedback” contained no more actionable information than a generic retry. Teams should inspect the payload of every failure signal before interpreting a top-line loop metric.

The author then instrumented the harness to report the failing input, expected output, and actual output. On the same 35 problems, the real-feedback loop solved 33, while the generic-feedback control remained at 32 and single-shot generation also remained at 32. The reported improvement came with about 2,500 additional input tokens across the run. The operational takeaway is concrete: a loop’s signal should expose the specific state transition that failed, not merely announce that a failure occurred.

Before trusting those numbers, the experiment also self-tested the grader against three cases: known-good code had to pass, known-bad code had to fail with an assertion error, and an infinite loop had to be terminated by a timeout. The scorer then passed all 75 MBPP+ reference solutions. This sequence matters because a broken grader can make a well-designed loop look effective; validate the evaluator before comparing agents.

What did the verifier miss?

The maker-checker verifier generated tests from the specification, ran those tests, and accepted a candidate only after a clean sweep. On 41 candidates—33 correct and 8 incorrect—the verifier falsely accepted 3 incorrect candidates, or 38%, while falsely rejecting 1 correct candidate, or 3%. The verifier therefore supplied stronger evidence than unconditional trust, but it did not establish correctness. Operators should treat a test-running checker as one gate in a chain, not as an oracle.

The reported false accepts came from three problems with ambiguous specifications. The generator and checker were the same model reading the same unclear sentence, so the checker wrote tests that encoded the same misunderstanding as the incorrect solution. The result is a warning for agent builders: independence is about information and assumptions, not merely placing a second model or a second stage after the first one.

Running tests still changed the quality of the evidence. A test result attached a concrete input, expected value, and actual value to the decision, while an opinion-based checker mostly expressed confidence or hesitation. The experiment’s figures show the trade-off: the test-running verifier rejected only 1 of 33 correct candidates, but it missed 3 of 8 wrong ones. Teams should preserve the low false-reject benefit while adding a hidden test suite, a fixed oracle, or a genuinely independent interpretation of ambiguous requirements.

Did combining the loop components improve results?

The combined system used the feedback loop first, sampled fresh candidates when needed, and let the verifier decide what could be submitted. On a held-out MBPP+ slice graded against hidden tests, single-shot generation solved 29 of 35 problems, while the loop-only system solved 34 of 35 and the loop-plus-verifier system also solved 34 of 35. The reported result attributes most of the gain to feedback-driven iteration, not to the verifier, so teams should measure each stage separately instead of crediting the whole pipeline.

The verifier ran once on the one problem the loop could not solve, and its first sampled candidate passed its own self-written tests while failing the hidden tests. The hidden runner caught that false accept because it used a source of truth the verifier could not see. This is the boundary that matters for production: a loop may improve completion rates while its internal verifier remains unable to detect a class of shared misunderstandings.

The author reports that the loop recovered five of six single-shot failures, used about ten extra API calls, and cost roughly thirteen cents for the combined stage. Those figures belong to this small benchmark and this implementation, not to AI coding agents in general. They are useful as an experimental baseline, not as a promise that every repository will gain 14.2 percentage points from the same design.

Where does the result stop?

The experiment applies to small, testable MBPP+ functions and does not establish how the same loop behaves in a large codebase with cross-file dependencies or parallel worktrees. The author also notes that the loop stops at verification and does not decide whether to auto-apply a change or escalate it to a human. Teams should therefore avoid treating the benchmark result as evidence that an agent can safely merge, deploy, or modify production systems without another control layer.

The experiment also identifies ambiguity as a failure source that better orchestration cannot solve by itself. When the generator and checker share the same interpretation of an unclear specification, more retries can reproduce the same mistake with better confidence. The next control should be a clearer requirement or an independent oracle, not simply another loop around the same assumption.

What should operators take from the experiment?

The experiment points to four practical checks for any AI coding loop:

• Self-test the grader first. Confirm that known-good, known-bad, and non-terminating programs produce the expected outcomes before collecting benchmark numbers.
• Run a generic-retry control. If real failure details do not beat “try again,” inspect the harness before changing the model or adding more retries.
• Measure both error types. Track false accepts and false rejects because a cautious checker can look safe while burying correct work in unnecessary retries.
• Keep an independent gate. Hidden tests, a fixed oracle, or human review should remain outside the model-generated evidence whenever the consequence of a false accept is material.

The most defensible conclusion is narrower than “loop engineering solves coding.” The reported benchmark improved when the agent received useful failure evidence, but the verifier still inherited ambiguity from the same specification and missed a hidden-test failure. A reliable loop therefore needs measurable progress, tested evaluation, and an independent stopping signal—not just more autonomous turns.