ai-boost/awesome-harness-engineering

Awesome Harness Engineering

Curated resources, patterns, and templates for building reliable AI agent harnesses.

Deutsch | English | Español | Français | 日本語 | 한국어 | Português | Русский | 中文

Harness engineering is the discipline of designing the scaffolding — context delivery, tool interfaces, planning artifacts, verification loops, memory systems, and sandboxes — that surrounds an AI agent and determines whether it succeeds or fails on real tasks.

This list focuses on the harness, not the model. Every component here exists because the model can't do it alone — and the best harnesses are designed knowing those components will become unnecessary as models improve.

---

Contents

• 📐 Foundations
• 🧩 Design Primitives
• 🔄 Agent Loop
• 🗺️ Planning & Task Decomposition
• 📦 Context Delivery & Compaction
• 🔧 Tool Design
• 🔌 Skills & MCP
• 🛡️ Permissions & Authorization
• 🧠 Memory & State
• ⚙️ Task Runners & Orchestration
• ✔️ Verification & CI Integration
• 👁️ Observability & Tracing
• 🐛 Debugging & Developer Experience
• 🧑‍💼 Human-in-the-Loop
• 🔍 Reference Implementations
• 🎓 Tutorials & Educational
• 🏭 Generators & Meta-Harnesses
• 🧪 Demo Harnesses
• 🗂️ Adjacent Collections
• 🔒 Security, Sandbox & Permissions
• ✅ Evals & Verification
• 📋 Templates
• 🏭 Production Infrastructure & Operations
• 📚 Related Awesome Lists
• 🤝 Contributing

Foundations

Canonical essays that define what harness engineering is and why it matters.

• Harness Engineering — OpenAI's framing of harness engineering as a discipline: how to design the scaffolding that lets Codex and similar agents operate reliably in an agent-first world.
• Unrolling the Codex Agent Loop — OpenAI's detailed breakdown of the Codex agent loop, exposing each harness component and where it can be improved.
• Run Long-Horizon Tasks with Codex — OpenAI's practice guide for long-horizon task planning: introduces Plan.md, Implement.md, Documentation.md as reusable harness artifacts.
• Building Effective Agents — Anthropic's foundational guide on agent architecture, covering when to use workflows vs. agents and how to compose primitives.
• Harness Design for Long-Running Application Development — Anthropic's engineering blog on designing harnesses for sustained, multi-session development tasks. Key insight: every harness component assumes the model can't do something; those assumptions expire.
• Writing Effective Tools for Agents — Anthropic's guide on tool interface design: naming, schemas, error surfaces, and the principle that tool design is agent UX.
• Beyond Permission Prompts — Anthropic on building structured permission and authorization systems into agent harnesses instead of relying on natural-language permission text.
• Demystifying Evals for AI Agents — Anthropic's framework for evaluating agent behavior: what to measure, how to build eval harnesses, and why unit-test-style evals fail for agents.
• What is an AI Agent? — IBM's definitional piece, useful for anchoring harness design decisions to a clear model of what an agent actually is.
• Agent Development Kit: Making it easy to build multi-agent applications — Google's announcement and design rationale for ADK: explains the multi-agent topology, tool registration model, and eval pipeline that shaped their framework. Complements the Anthropic/OpenAI framing with Google's production perspective.
• Harness Engineering — Martin Fowler's synthesis of what harness engineering practice looks like: three interlocking systems — context engineering (curating what the agent knows), architectural constraints (deterministic linters and structural tests), and entropy management (periodic agents that repair documentation drift). The "humans on the loop" framing — harness engineers who design and maintain agent environments rather than inspecting individual outputs — is the clearest conceptual map of what the discipline actually entails.
• The Anatomy of an Agent Harness — LangChain's structural breakdown of the five primitives that compose a harness: filesystem (durable state + agent collaboration surface), code execution (autonomous problem-solving without pre-designed solutions), sandbox (isolation + verification), memory (cross-session persistence), and context management (compaction against "context rot"). The co-evolution warning — models trained with specific harnesses can become overfitted to those designs — explains why harness architecture choices have lasting consequences beyond the immediate task.
• Building AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned — The first systematic practitioner paper on terminal-native coding agent harness design: eager-construction scaffolding (pre-build all components before the first message to eliminate first-call latency and race conditions), compound multi-model architecture (different model instances for execution, reasoning, critique, and vision tasks), 5-layer defense-in-depth safety, and schema-filtered planning subagents (enforce behavioral constraints via tool schema rather than runtime permission checks). The five lessons distilled from building OpenDev apply to any server-side agent harness.
• Natural-Language Agent Harnesses — Proposes externalizing agent control logic as portable natural-language artifacts (NLAHs) executed by a shared Intelligent Harness Runtime, enabling harness design to be studied, transferred, and reproduced rather than buried in bespoke controller code. Directly addresses the root cause of harness fragility: control logic scattered across framework defaults and hard-coded controller logic that can't be inspected, versioned, or transferred.
• Ranking Engineer Agent (REA): Meta's Autonomous AI System for Ads Ranking — Meta's production harness for multi-day ML pipeline automation with hibernate-and-wake checkpointing for resuming interrupted 6-hour tasks without losing context. Demonstrates harness design for scientific workflows where individual turns can exceed model context limits but the overall pipeline must maintain coherence across days.
• Supercharge Your AI Agents: The New ADK Integrations Ecosystem — Google's 2026 update to Agent Development Kit expanding the ecosystem integrations (Hugging Face, GitHub, Daytona, Notion, etc.) and providing reference patterns for how orchestration harnesses wire external services without losing determinism or state coherence.
• 2026 Agentic Coding Trends Report — Anthropic's industry benchmark identifying infrastructure configuration as a first-class optimization variable: harness setup alone can swing benchmarks by 5+ percentage points. Documents the shift from single-agent to orchestrated multi-agent teams and introduces the "agentic engineering platform" category, bridging the gap between agent frameworks and production deployment infrastructure.
• How We Build Azure SRE Agent with Agentic Workflows — Architecture walkthrough of Microsoft's agent that has handled 35,000+ production incidents autonomously, reducing Azure App Service time-to-mitigation from 40.5 hours to 3 minutes. Documents the integration of MCP tools, telemetry, code repositories, and incident management platforms into a single agent harness with human-in-the-loop governance. The most data-backed production harness case study published in 2026.
• Context Engineering for Reliable AI Agents: Lessons from Building Azure SRE Agent — Microsoft's account of shifting from 100+ bespoke tools and a prescriptive prompt to a filesystem-based context engineering system for their SRE agent. Key finding: exposing everything (source code, runbooks, query schemas, past investigation notes) as files and letting the agent use read_file, grep, find, and shell outperformed specialized tooling — "Intent Met" score rose from 45% to 75% on novel incidents.
• Harness Engineering: Structured Workflows for AI-Assisted Development — Red Hat's enterprise perspective on harness engineering (April 7, 2026): AI writes better code when you design the environment it works in. Emphasizes structured context over free-form tickets, expanding the agent's toolbox through MCP integrations (CI status, deployment logs, runtime metrics) as real data sources, and a four-pillar model (vibes, specs, skills, agents) for organizing how humans and agents collaborate.
• Harness engineering for coding agent users — Birgitta Böckeler's systematic mental model (April 2026) for coding-agent harnesses, framing them as feedforward guides plus feedback sensors that self-correct before output reaches human eyes. Distinguishes computational controls (linters, tests) from inferential ones (LLM-as-judge), and argues that harnessability should become a first-class criterion in technology and architecture decisions.
• A Practical Guide to Building AI Agents — OpenAI's April 2026 comprehensive guide distilling production deployment patterns into actionable best practices: single-agent vs. multi-agent orchestration (manager vs. decentralized handoffs), tool design for many-to-many agent-tool relationships, and layered guardrail patterns combining input validation, output filtering, tool-risk ratings, and human-intervention triggers.
• An Update on Recent Claude Code Quality Reports — Anthropic's transparent April 2026 postmortem tracing Claude Code quality degradation to three independent harness-level changes: a default reasoning-effort downgrade, a caching-optimization bug that continuously dropped thinking history from stale sessions, and an overly aggressive verbosity-limiting system prompt. Essential reading for understanding how seemingly minor harness adjustments — prompt wording, cache headers, and default parameters — can compound into visible agent regressions, and for the rigorous diagnostic process required to isolate them.
• Agent Harness Design: 3 Patterns for Harnessing Claude's Intelligence — Anthropic's April 2026 design guide distilling harness engineering into three actionable patterns: build on tools Claude already knows, remove harness assumptions as capabilities improve, and set UX/cost/safety boundaries carefully. A practical complement to the "agent = model + harness" framing that helps teams decide what scaffolding to keep, add, or remove over time.
• Code as Agent Harness — May 2026 survey framing code as the basis for agent infrastructure rather than merely output: it unifies harness interface, mechanisms, and multi-agent scaling through shared code artifacts, and surfaces open challenges from verification under incomplete feedback to regression-free improvement.
• Harness Engineering: How to Build Reliable AI Agents by Engineering the System, Not the Model — deepset's May 2026 synthesis of agent reliability as a harness problem: a failure-classification framework (context, constraint, verification, planning failures) that maps each failure mode to the right harness component, and a concrete demonstration that harness-only changes can move agents 20+ ranking positions without swapping the model.
• What makes a harness a harness: necessary and sufficient conditions for an agent harness — June 2026 constitutive definition of an agent harness as a runtime layer with four necessary and sufficient elements: an agent loop, a tool interface, context management, and control mechanisms. Applied to Claude Code, Codex CLI, Aider, Cline, OpenHands, and SWE-agent, it provides a rigorous inclusion test for distinguishing harnesses from generators, guardrails, or plain tool wrappers.
• Architectural Design Decisions in AI Agent Harnesses — April 2026 empirical study of 70 public agent systems across five recurring dimensions (subagent architecture, context management, tool systems, safety mechanisms, orchestration) that synthesizes five architectural patterns. The comparative research package turns harness selection from a framework popularity contest into a reasoned comparison of design trade-offs.
• RUCAIBox/awesome-agent-harness — RUCAIBox's survey paper and curated reading list on Agent Systems with Harness Engineering, mapping harness design across agent workflows, memory systems, skill libraries, and multi-agent orchestration with 500+ references. The clearest academic complement to vendor-specific harness engineering posts.
• Tuning the harness, not the model: a Nemotron 3 Ultra playbook — LangChain's July 2026 playbook showing how harness-only tuning brought Nemotron 3 Ultra within one point of Opus 4.8 on Deep Agents at roughly one-tenth the cost ($4.48 vs $43.48). The clearest recent demonstration that evals are the training data for harness work and that fit — not raw model capability — determines how much quality reaches the task.
• lopopolo/harness-engineering — Ryan Lopopolo's anthology, field guide, and agent context bundle for harness engineering: it reframes the harness as the environment that carries an organization's nonfunctional requirements, with reusable AGENTS.md/CLAUDE.md artifacts, playbooks, evals, and domain modeling docs. The most systematic open-source synthesis of how to make organizational judgment cumulative across agent-maintained repositories.
• Harness Engineering for Self-Improvement — Lilian Weng's July 2026 synthesis framing the harness as the deployment layer of recursive self-improvement: three design patterns (workflow automation, filesystem as persistent memory, sub-agents/backend jobs), a case study of the tool surface stabilized across Claude Code, Codex, OpenCode, and Cursor, and a map of self-improving-harness research from STOP through Meta-Harness, ADAS, AFlow, and AlphaEvolve. The operating-system analogy — a harness should encapsulate complicated logic behind a simple interface — and the argument that code is the universal language for searching harness design space make this the best single survey connecting everyday harness design to the meta-harness optimization literature.

Design Primitives

Harness components organized by the problem they solve, not by vendor.

Agent Loop

• ReAct: Synergizing Reasoning and Acting in Language Models — The foundational paper defining the Thought/Action/Observation loop structure that underlies virtually every agent harness. Required reading for understanding why the loop is structured the way it is and where each harness component maps onto the reasoning-acting cycle.
• Unrolling the Codex Agent Loop — The canonical decomposition of what happens inside one agent loop iteration: observe, plan, act, verify.
• LangGraph — Low Level Concepts — Models the agent loop explicitly as a directed graph with typed state, conditional edges, and checkpointing. The most concrete engineering treatment of loop control flow: how to implement termination conditions, branch on tool results, and persist mid-loop state for resumption.
• Unlocking the Codex Harness: How We Built the App Server — OpenAI's engineering deep-dive into the Item/Turn/Thread protocol (JSON-RPC/JSONL over stdio) that exposes the Codex harness to every client surface. The most direct first-party account of why approval flows, streaming diffs, and thread persistence demand a purpose-built protocol — and why MCP's tool-oriented model proved insufficient for these requirements.
• Hooks – Codex — OpenAI's lifecycle-hook framework for Codex: inject deterministic scripts at SessionStart, PreToolUse, PostToolUse, and other loop events to enforce guardrails, audit actions, and customize agent behavior without relying on prompt-level trust. A concrete reference for programmable harness governance.
• Extended Thinking — Claude API Docs — The harness-critical reference for integrating extended thinking into agent loops: budget_tokens controls reasoning depth per turn, thinking blocks must be preserved when passing tool results back (omitting them silently breaks multi-step reasoning), and thinking mode cannot change mid-turn. Essential before wiring extended thinking into any tool-use loop.
• Getting started with loops — Anthropic's June 2026 practical taxonomy of agent loops: turn-based, goal-based (/goal), time-based (/loop, /schedule), and proactive loops. The framework for matching loop primitive to task shape — and the emphasis on deterministic stop conditions and token budgets — makes it a concise reference for choosing the right loop abstraction instead of defaulting to a single conversational turn cycle.
• Improving Deep Agents with Harness Engineering — LangChain's case study showing harness-only changes moved their coding agent from rank 30 to top 5 on Terminal Bench 2.0 with no model swap: structured verification loops, context injection (directory maps + time budget warnings), loop-detection middleware, and a "reasoning sandwich" concentrating maximum thinking at planning and verification phases. The most concrete published demonstration that harness design is the primary performance lever, not model capability.
• Loop Engineering — Practical design system for agent loops with seven production patterns, cross-tool starter kits, and CLI tools that score readiness, scaffold state, estimate cost, detect drift, and isolate worktrees. The clearest open-source resource for moving from one-off prompting to durable, observable agent loops.
• Life-Harness — Official implementation of a lifecycle-aware runtime harness that improves frozen LLM agents by adapting the model-environment interface across four layers: environment contract, procedural skills, action realization, and trajectory regulation. The key result is that harness-side adaptation transfers across 18 model backbones, proving that many agent failures are interface mismatches rather than reasoning deficits.
• How Middleware Lets You Customize Your Agent Harness — Introduces AgentMiddleware: six composable hooks (before_agent, before_model, wrap_model_call, wrap_tool_call, after_model, after_agent) that intercept every stage of the agent loop. Enables deterministic policy enforcement (PII redaction that can't be trusted to prompts), dynamic tool injection, mid-task model swapping, and production patterns (retry, fallback, HITL interrupts) without modifying core agent logic — the reference des