lessonweaver Turns AI Agent Mistakes into Reviewed Skills

lessonweaver is a deterministic, human-gated way to turn recurring AI-agent failures into reviewed instruction files instead of letting an agent rewrite its own rules. The source story describes a Python tool that mines execution traces, routes candidate lessons through structured review, applies a promotion gate, and exports approved guidance for surfaces such as AGENTS.md, Claude, and GitHub Copilot. For teams running coding agents, the takeaway is simple: capture corrections as evidence, but require a person to decide which corrections become durable context. The distinction between an agent and a chatbot is covered in What Is an AI Agent, while Skills to Build AI Agents and AI Agent Mistakes Beginners Should Avoid give broader background on instruction-driven agent capabilities and failure prevention.

Definition: lessonweaver is a trace-to-guidance tool that detects candidate lessons and publishes only human-reviewed, lint-gated instruction artifacts.

Example: A pull-request agent that approves a change after reading only the title and description can produce a candidate lesson requiring diff inspection before a review conclusion.

Key takeaway: lessonweaver keeps learning governed: a failure can become a candidate, but it does not become an active skill automatically.

Business impact: Operators can reduce repeated agent mistakes without accepting an opaque self-editing loop that changes production guidance without an audit trail.

Why do AI agents repeat corrected mistakes?

AI agents repeat corrected mistakes when a human correction stays in the conversation or ticket but never reaches durable runtime context. The source uses a coding-agent review that skipped the diff as its concrete example: a human fixes the review once, but another run can make the same error because the correction was not converted into reusable guidance. Teams should treat repeated corrections as evidence to capture, not as proof that an agent has learned.

The underlying trade-off is between unsafe self-editing and slow manual maintenance. An agent that rewrites its own instructions can reinforce a bad habit or drift without a clear audit trail, while a human who manually curates every AGENTS.md rule may leave recurring lessons undocumented. lessonweaver occupies the middle: deterministic detection proposes a lesson, structured human review decides whether it is valid, and a governed export creates a change that can be inspected before use.

How does lessonweaver govern a lesson?

lessonweaver governs a lesson through five explicit stages: detect, interview, answer, approve, and export. The sequence matters because each stage separates evidence collection from judgment: the detector finds a signal, the interview creates review questions, the reviewer records answers, approval promotes the lesson, and export writes a target-specific artifact. Teams adopting the workflow should preserve those boundaries so a candidate is never mistaken for an approved rule.

1. Detect: lessonweaver scans a documented trace for signals such as a human correction, a failed evaluation, an error followed by a retry, or an out-of-order workflow step.
2. Interview: lessonweaver generates structured questions about the candidate lesson.
3. Answer: a human records the decision and the operational lesson that should be retained.
4. Approve: lessonweaver blocks promotion when required review questions are incomplete unless an explicit, auditable override is used.
5. Export: lessonweaver emits a diff-first instruction artifact for the selected agent surface.

The approval stage is the key control because lessonweaver does not treat detection as truth. The source describes incomplete reviews being rejected by the approval command, with an override recorded when a team deliberately allows the shortcut. That makes the governance decision visible: teams can distinguish a fully reviewed lesson from an exception instead of treating every generated candidate as equally trustworthy.

What makes lessonweaver different from agent self-editing?

lessonweaver differs from agent self-editing by keeping the core deterministic and human-gated. The source describes no LLM in the loop for detection, review, linting, or retrieval, so lessonweaver is not asking a second model to invent a policy from the first model's failure. Teams that need predictable change control should prefer this separation when instruction files can affect production code, security checks, or other high-consequence workflows.

The comparison is not an argument that lessonweaver removes operational risk. Its conservative, rule-based detector can miss subtle or novel failure modes, and its trace redaction is described as best-effort. Operators should use lessonweaver as a governed learning loop alongside evaluation and observability, not as a replacement for either one.

Which instruction surfaces can lessonweaver target?

lessonweaver targets the instruction surfaces teams already maintain instead of creating a proprietary memory silo. The source lists AGENTS.md fragments, Claude skills and CLAUDE.md rules, GitHub Copilot instructions, Codex skill directories, and generic runtime snippets as export targets. Teams should choose the surface that their agent actually loads, then review the generated diff like any other change to operational documentation.

lessonweaver's runtime loading uses lexical retrieval with a character budget. The source describes lessonweaver selecting approved guidance for the current task rather than dumping every lesson into every context window. For operators, lessonweaver makes the practical rule clear: keep lessons narrow and evidence-linked so the selected instruction set is easier to review and less likely to compete with unrelated context.

What should teams watch before adopting it?

Teams should evaluate lessonweaver as early software that depends on disciplined traces, not as a turnkey memory layer. The source identifies early-alpha maturity, a documented trace-format requirement, conservative detection, best-effort redaction, and clear boundaries against agent frameworks, evaluation runners, and observability products. A sensible pilot starts with non-sensitive traces, measures which known corrections are detected, and checks every exported rule in the target agent before wider rollout.

lessonweaver's broader lesson is about the shape of agent memory, not just one Python repository. lessonweaver shows why durable guidance needs provenance, a reviewer, a scope, and a reversible export path; teams considering another detector or instruction loader should preserve those controls by recording the failure, reviewing the lesson, linting the artifact, and verifying that the next agent run receives the approved guidance.