Harness engineering is the discipline of designing the scaffolding — context delivery, tool interfaces, planning artifacts, verification loops, memory systems, and sandboxes — that surrounds an AI agent and determines whether it succeeds or fails on real tasks.
This list focuses on the harness, not the model. Every component here exists because the model can't do it alone — and the best harnesses are designed knowing those components will become unnecessary as models improve.
Contents
Foundations
Canonical essays that define what harness engineering is and why it matters.
- Harness Engineering — OpenAI's framing of harness engineering as a discipline: how to design the scaffolding that lets Codex and similar agents operate reliably in an agent-first world.
- Unrolling the Codex Agent Loop — OpenAI's detailed breakdown of the Codex agent loop, exposing each harness component and where it can be improved.
- Run Long-Horizon Tasks with Codex — OpenAI's practice guide for long-horizon task planning: introduces Plan.md, Implement.md, Documentation.md as reusable harness artifacts.
- Building Effective Agents — Anthropic's foundational guide on agent architecture, covering when to use workflows vs. agents and how to compose primitives.
- Harness Design for Long-Running Application Development — Anthropic's engineering blog on designing harnesses for sustained, multi-session development tasks. Key insight: every harness component assumes the model can't do something; those assumptions expire.
- Writing Effective Tools for Agents — Anthropic's guide on tool interface design: naming, schemas, error surfaces, and the principle that tool design is agent UX.
- Beyond Permission Prompts — Anthropic on building structured permission and authorization systems into agent harnesses instead of relying on natural-language permission text.
- Demystifying Evals for AI Agents — Anthropic's framework for evaluating agent behavior: what to measure, how to build eval harnesses, and why unit-test-style evals fail for agents.
Design Primitives
Harness components organized by the problem they solve, not by vendor.
Agent Loop
- ReAct: Synergizing Reasoning and Acting in Language Models — The foundational paper defining the Thought/Action/Observation loop structure that underlies virtually every agent harness. Required reading for understanding why the loop is structured the way it is and where each harness component maps onto the reasoning-acting cycle.
- Unrolling the Codex Agent Loop — The canonical decomposition of what happens inside one agent loop iteration: observe, plan, act, verify.
- LangGraph — Low Level Concepts — Models the agent loop explicitly as a directed graph with typed state, conditional edges, and checkpointing. The most concrete engineering treatment of loop control flow: how to implement termination conditions, branch on tool results, and persist mid-loop state for resumption.
- Unlocking the Codex Harness: How We Built the App Server — OpenAI's engineering deep-dive into the Item/Turn/Thread protocol (JSON-RPC/JSONL over stdio) that exposes the Codex harness to every client surface. The most direct first-party account of why approval flows, streaming diffs, and thread persistence demand a purpose-built protocol — and why MCP's tool-oriented model proved insufficient for these requirements.
- Hooks – Codex — OpenAI's lifecycle-hook framework for Codex: inject deterministic scripts at
SessionStart, PreToolUse, PostToolUse, and other loop events to enforce guardrails, audit actions, and customize agent behavior without relying on prompt-level trust. A concrete reference for programmable harness governance.
- Extended Thinking — Claude API Docs — The harness-critical reference for integrating extended thinking into agent loops: controls reasoning depth per turn, thinking blocks when passing tool results back (omitting them silently breaks multi-step reasoning), and thinking mode cannot change mid-turn. Essential before wiring extended thinking into any tool-use loop.
Planning & Task Decomposition
- Run Long-Horizon Tasks with Codex — Introduces milestone-based planning artifacts (Plan.md, Implement.md) as harness-level state.
- Harness Design for Long-Running Application Development — Multi-session planning, progress tracking, and the role of persistent planning documents.
- Plan-and-Execute Agents — The canonical engineering write-up separating planning from execution as distinct harness layers: a planner LLM generates the step list once; an executor agent works through it, replanning only when needed. Defines the pattern that most modern task-decomposition harnesses follow.
- microsoft/TaskWeaver — Code-first task decomposition framework with a planner/executor split and a plugin system for injecting domain knowledge into the planning layer. The most complete reference implementation of plan-then-execute with stateful task tracking.
- LATS: Language Agent Tree Search — Unifies reasoning, acting, and planning via Monte Carlo Tree Search over agent trajectories. Directly informs harness design: external tool feedback as tree-search signals, trajectory backtracking on failure, and depth-bounded exploration make this the most actionable planning research for harnesses with real environment interaction.
- Agyn: A Multi-Agent System for Team-Based Autonomous Software Engineering — Demonstrates specialized harness patterns for coordinating heterogeneous agent teams (planner, coder, reviewer, executor) on software engineering tasks. Shows how role-specific agents with different model sizes and tool access produce better outcomes than single-agent approaches, with concrete metrics on task decomposition effectiveness.
Context Delivery & Compaction
- Harness Engineering — How to structure context windows for agents: what to include, what to exclude, and how context shape affects agent behavior.
- Effective Context Engineering for AI Agents — Anthropic's systematic guide to managing the full context state—system prompts, tools, MCP, and message history—as a finite, curated resource. Reframes harness design as "what configuration of context produces the desired behavior?" rather than just prompt wording.
- Compaction — Claude API Docs — Anthropic's reference for server-side context compaction: automatically summarizes older context when approaching the window limit. Reduced token consumption by 84% in a 100-turn web search eval while allowing agents to complete workflows that would otherwise hit context limits.
- LLMLingua — Microsoft Research's prompt compression toolkit (up to 20x compression, minimal performance loss) that can be embedded as a preprocessing step in the context delivery layer. LLMLingua-2 adds 3–6x speed gains, making it viable for latency-sensitive agent loops.
- Prompt Caching — Claude API Docs — The most effective harness-level cost lever: cache repeated system prompts, tool definitions, and long documents across requests. Explains where to place
cache_control breakpoints for maximum reuse across multi-turn agent sessions.
- Autonomous Context Compression — Shifts context compression from harness-controlled (compacting at a fixed token threshold) to agent-controlled: agents call a dedicated tool to trigger compression when strategically appropriate — between tasks or before consuming large inputs. Eliminates the failure mode where reactive-at-limit compaction interrupts agents mid-subtask and corrupts in-flight reasoning state.
Tool Design
- Writing Effective Tools for Agents — Tool naming, schema design, error messages, and return value conventions that make agents more reliable.
- Tool Use — Claude API Docs — Authoritative reference for client vs. server tool execution models, strict schema enforcement, and tool_result error signaling. The distinction between client-side and server-side tool execution is a foundational harness architecture decision.
- Function Calling — OpenAI Docs — Defines the de facto industry-standard JSON Schema conventions for tool definitions and parallel function calling. Essential reading before designing a tool interface that needs to work across multiple models.
- Tool Annotations as Risk Vocabulary — The MCP team's definitive post on the four tool annotation hints (
readOnlyHint, destructiveHint, idempotentHint, openWorldHint) as inputs to harness permission decisions, not enforced contracts. The "lethal trifecta" — private data access + untrusted content exposure + external communication — is the most actionable framing for why single-tool safety analysis misses the risk that emerges from tool combinations.
- outlines — Constrains token sampling via regex/CFG/JSON Schema at the decoding layer, guaranteeing structured output without model fine-tuning. The right solution when you need OpenAI Structured Outputs-equivalent reliability from a locally deployed or open-weight model.
Skills & MCP
- Model Context Protocol — Anthropic's open protocol for connecting agents to external tools, data sources, and services in a standardized way.
- modelcontextprotocol/servers — Anthropic's official reference MCP server implementations (GitHub, Slack, Postgres, Puppeteer, etc.). The authoritative source for understanding correct MCP server structure before building your own.
- microsoft/playwright-mcp — Browser automation via accessibility tree snapshots rather than screenshots, dramatically reducing token cost. The canonical example of structured tool output design in an MCP server.
- Chrome DevTools MCP — Official Google MCP server that exposes live Chrome debugging surfaces — network analysis, performance profiling, console messages, memory snapshots, and Lighthouse audits — as structured agent tools. The clearest reference for turning browser inspection into a first-class tool interface rather than relying solely on screenshot-driven automation.
- agent-device — MCP-native control layer for iOS and Android devices: snapshots, semantic targeting, typed client access, diagnostics, and replayable workflows. Fills a critical gap in the mobile-agent harness stack — most tool design assumes desktop or browser surfaces, but real-world agents increasingly need to interact with native mobile apps.
Permissions & Authorization
- Beyond Permission Prompts — Structured authorization patterns for agents: how to give agents the right permissions without relying on prompt-level trust.
- OWASP LLM06:2025 — Excessive Agency — OWASP's authoritative definition of the "excessive agency" risk: over-provisioned functions, unnecessary permissions, and missing approval mechanisms. The standard checklist for auditing harness permission scope against principle of least privilege.
- GitHub Enterprise — Governing Agents — April 2026 GitHub official guide for enterprise agent governance: MCP server registry curation with ruleset-protected configurations, agent environment standardization via
copilot-setup-steps.yml, ephemeral runner enforcement, and cloud-agent firewall allowlisting. The most concrete published reference for governing agent fleets at scale without creating bottlenecks.
- Claude Code Auto Mode: A Safer Way to Skip Permissions — Anthropic's engineering post on replacing approval fatigue (users approve 93% of prompts, making approvals meaningless) with a two-stage classifier: fast single-token gate first, chain-of-thought reasoning only on flagged actions. The design decisions — stripping assistant messages to prevent the agent from rationalizing dangerous actions, deny-and-continue recovery instead of halt — are the reference design for safe-by-default headless agent permissions.
- Claude Agent SDK — Configure Permissions — The most concrete reference for harness permission architecture: five-layer evaluation order (hooks → deny rules → permission mode → allow rules → canUseTool),
allowedTools/disallowedTools declarative scoping, and four permission modes including (deny-by-default for headless agents). The subagent inheritance warning for alone is worth reading before any multi-agent deployment.
Memory & State
- Building Effective Agents — Covers in-context, external, and procedural memory patterns as harness-level concerns.
- Letta (MemGPT) — The reference architecture for stateful agents: three-tier memory (core / archival / recall) maps directly to harness state management design. Their agent loop redesign post is the most thorough public analysis of how memory structure shapes the harness.
- mem0 — Drop-in universal memory layer (YC-backed, AWS Agent SDK's exclusive memory provider) that handles cross-session retention without custom harness-level state management code. Lowest integration cost for production-grade persistent memory.
- Stash — Self-hosted persistent memory layer with an 8-stage consolidation pipeline (episodes → facts → relationships → patterns) and built-in MCP server. The critical gap it fills: production-grade cross-session memory without cloud dependencies or complex infrastructure — a single Docker Compose gives you Postgres, pgvector, and background consolidation.
- TencentDB-Agent-Memory — Tencent's fully local agent memory system with a 4-tier progressive pipeline (Conversation → Atom → Scenario → Persona) and symbolic short-term memory via Mermaid canvases. The benchmark data is striking: 61% token reduction and 51% relative pass-rate improvement on long-horizon tasks, demonstrating that hierarchical memory architecture outperforms flat vector stores for coding agents.
Task Runners & Orchestration
- Harness Engineering — How task runners fit into the harness: queueing, parallelism, and progress reporting.
- Building a C Compiler with a Team of Parallel Claudes — Anthropic's account of coordinating 16 Claude instances in parallel on a shared git repo without a central orchestrator: agents claim tasks via files in
current_tasks/, git forces collision resolution naturally, and a continuous restart loop spawns fresh sessions that resume where predecessors left off. Key harness lesson: verbose test output pollutes agent context — the feedback loop must emit only a few summary lines, log detail to file.
- LiteLLM — Unified proxy and SDK that routes to 100+ LLM providers behind a single OpenAI-compatible interface, with a Router handling retry/fallback across deployments, per-project cost and rate-limit tracking, and OTEL callback integrations. The right infrastructure layer when your harness needs provider resilience (automatic failover on 429/500 errors), budget guardrails, or the ability to swap models without touching orchestration code.
- LangGraph — Graph-based state machine framework for multi-agent harnesses: models supervisor/subagent topologies, error-recovery branches, and checkpoint persistence as first-class primitives. The most widely adopted harness orchestration layer in production.
- OpenAI Agents SDK — Lightweight multi-agent framework built around handoffs and guardrails; the production successor to Swarm. Complements LangGraph for harnesses where delegation patterns are simpler than full graph orchestration.
Verification & CI Integration
- Demystifying Evals for AI Agents — How to build verification into the harness loop, not just as a post-hoc eval.
- promptfoo — YAML-driven LLM testing framework with LLM-as-judge, assertion DSL, and native CI integration. The most practical tool for adding agent output regression tests to a PR pipeline without writing a test harness from scratch.
- AgentBench — Multi-environment agent benchmark (OS, DB, web, code) with a structured eval pipeline. Worth studying for its environment isolation design and task definition format when building custom eval environments for your harness.
- Testing Agent Skills Systematically with Evals — OpenAI's framework for skill regression testing: four eval dimensions (outcome, process, style, efficiency goals), JSONL trace capture for deterministic checks (command sequences, token budgets, repo cleanliness), then rubric-based grading only where deterministic checks don't suffice. The layering principle — add expensive LLM-as-judge checks only where they reduce meaningful risk — is the most actionable published guide to CI pipelines for agent skills that don't collapse under eval cost.
- Agent Evaluation Readiness Checklist — A 33-item checklist covering the full evaluation lifecycle: error taxonomy, three-level granularity (single-step → trace → multi-turn thread), grader specialization, and CI integration. Key insight: capability evals (low pass rate, improvement target) and regression evals (near-100%, protection target) must be separated — mixing them produces wrong prioritization decisions.
- Evaluating Skills — LangChain's methodology for benchmarking agent skills in Docker-sandboxed environments. Key empirical findings: Claude Code achieved 82% task completion with curated skills vs. 9% without, and consolidating to ≤12 skills improved accuracy over sprawling skill sets. The baseline-vs-skills comparison design with bugfix tasks and clear outcome metrics is the template for systematic skill coverage testing.
Observability & Tracing
- OpenLLMetry — OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness.
- Arize Phoenix — Self-hostable trace UI and eval runtime for agent workflows. Lets harness engineers audit and replay every reasoning step and tool call offline, without sending data to a third-party cloud.
- Opik — Comet's open-source AI observability and evaluation platform: deep tracing of LLM calls, conversation logging, and agent activity, plus built-in eval metrics, prompt versioning, guardrails, and the Opik Agent Optimizer. Worth including because it unifies observability, verification, and optimization in one self-hostable stack rather than stitching together separate tools.
- Langfuse — The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint.
- Weights & Biases Weave — W&B's tracing and eval layer purpose-built for agent workflows: automatic call graph capture, dataset versioning, and LLM-as-judge evals that integrate directly with the wandb experiment tracking ecosystem.
Debugging & Developer Experience
- AgentOps — Open-source agent engineering platform (YC W24) with session replay, cost tracking, and failure detection across 10+ frameworks including CrewAI, LangGraph, and OpenAI Agents SDK. The step-by-step execution graph and cross-session metrics make it the most practical debugging layer for multi-agent systems in production.
- claude-devtools — February 2026 open-source DevTools for Claude Code that reconstructs hidden session internals from local logs: per-turn token attribution across 7 context categories, full subagent execution trees with cost breakdowns, and syntax-highlighted diffs for every tool call. Essential because Claude Code's default UI deliberately collapses tool details and thinking steps — this tool restores the visibility harness engineers need to debug context pressure, subagent delegation, and token budget leaks.
- Syncause/debug-skill — April 2026 agent debugging skill that stops guesswork with runtime evidence. Uses background tracing (Runtime Facts) to capture the exact execution path leading to failures, then constrains the agent to cite specific data points (stack traces, variable snapshots) before proposing fixes. Moves agent debugging from "patch and pray" to evidence-based repair with reviewable results.
- AgentTrace: Causal Graph Tracing for Root Cause Analysis in Multi-Agent Systems — March 2026 framework that localizes root causes in multi-agent execution traces using causal graph analysis rather than LLM inference. Processes traces in 0.12 seconds (69× faster than LLM-based analysis) with 93.6–95.8% accuracy across 550 synthetic failure scenarios. Distinguishes root causes from downstream symptom propagation — the key capability missing from most trace-inspection debugging workflows.
Human-in-the-Loop
- aws-samples/sample-human-in-the-loop-patterns — March 2026 AWS reference implementation demonstrating four distinct HITL patterns for sensitive agent tool calls: Hook System (centralized blanket policy), Tool Context (per-tool fine-grained), Step Functions (async third-party approval via SNS), and MCP Elicitation (protocol-native real-time interactive approval). The most concrete production guide for choosing the right HITL architecture based on approval latency, trust boundaries, and integration constraints.
- Dify Human-in-the-Loop Node — February 2026 release making human oversight a native workflow primitive: suspend execution at critical decision points, expose review-and-edit UI mid-flow, and route subsequent execution based on human action (approve/reject/escalate). Demonstrates how HITL transitions from bolt-on approval gates to first-class execution-graph nodes with stateful pause/resume backed by Celery workers and Redis Pub/Sub.
- HITL Protocol — Open standard (v0.8, February 2026) for human decisions in agent workflows: HTTP 202 + review URL pattern connecting services, agents, and humans across any messaging channel. No SDK required — ~15 lines of code for agents, reference implementations in Express/Hono/Next.js/FastAPI for services, and 13 end-to-end flows including escalation and hybrid approval.
- LangGraph — Human-in-the-Loop Concepts — Systematic treatment of interrupt, breakpoint, and approve patterns: how to pause an agent mid-loop, persist state, and resume after human review. Directly addresses the harness engineering challenge of inserting human gates into long-running workflows.
- AutoGen — Human-in-the-Loop — Explains
human_input_mode (NEVER / TERMINATE / ALWAYS) and the UserProxyAgent as an approval gate. The most concrete implementation reference for adding human review nodes to a multi-agent conversation harness.
Reference Implementations
Real repositories worth studying — each with a note on why it's worth your time.
Tutorials & Educational
- Learn Harness Engineering — A project-based course on designing the environments, state, verification, and control systems that make Codex and Claude Code reliable. The most approachable public curriculum for learning harness engineering from first principles — each module builds a working artifact rather than summarizing concepts.
- ML6 x AISO Agent Workshop — February 2026 hands-on workshop building an AI agent from scratch with Google's Agent Development Kit (ADK) in 3 hours. Five milestones with a built-in benchmark that tracks progress from ~19% (base agent) to ~81% (with web search, PDF reader, and calculator tools). The clearest public tutorial for understanding how tool access directly translates to capability gains.
- mastra-ai/workshop-mastracode — February 2026 workshop by Mastra's founders dissecting every layer of an open-source AI coding agent: stateful/resumable harness, dynamic prompt composition, workspace sandboxing, memory compaction, HITL steering, event protocols, and cost tracking. The 11-topic curriculum is the most complete public walkthrough of production coding-agent harness internals.
- Building Governed AI Agents — OpenAI's February 2026 cookbook building a complete multi-agent governance system from scratch: policy-as-code guardrails, OpenAI Traces for full observability, eval-driven design, and a distributable governance package. The most concrete first-party tutorial for making governance part of core infrastructure from day one.
- anthropics/claude-cookbooks — Anthropic's official notebook collection covering orchestrator-worker patterns, parallel tool calling, programmatic tool calling (PTC), context compaction, and Agent SDK examples. The directory is the reference implementation of every orchestration pattern described in .
Generators & Meta-Harnesses
- everything-claude-code — Anthropic Hackathon Winner (140K+ stars). The agent harness performance optimization system: skills, instincts, memory optimization, continuous learning, security scanning, and research-first development. Production-ready agents, skills, hooks, rules, and MCP configurations evolved over 10+ months of intensive daily use building real products. Works across Claude Code, Codex, Cursor, OpenCode, and Gemini.
- Claude Agent SDK — Anthropic's official SDK that exposes Claude Code's entire harness as a programmable API: built-in tool execution loop,
PreToolUse/PostToolUse hooks for interception, subagent definitions, allowedTools permission control, and session resumption. The highest-leverage starting point for building a production harness — you inherit the entire tool execution layer rather than implementing it. 
- revfactory/harness — A meta-skill that generates domain-specific agent teams and the skills they use. Good example of harness-as-code, where the harness itself is produced by an agent.
- raphaelchristi/harness-evolver — March 2026 Claude Code plugin that autonomously evolves LLM agent harnesses using multi-agent proposers in isolated git worktrees, LangSmith-backed evaluation, and regression guards. Iterates on prompts, routing, retrieval, and orchestration code based on full-trace counterfactual diagnosis. The most practical published implementation of the Meta-Harness outer-loop optimization paradigm.
Demo Harnesses
- Anthropic Computer Use Demo — Anthropic's reference harness for the screenshot-action loop: defines the
screenshot, bash, and text_editor tool interface that makes desktop/browser control work. Essential reading before building any harness where the agent's primary sensory input is a rendered screen rather than structured API responses. 
- coleam00/your-claude-engineer — Agent harness with Slack, GitHub, and Linear integrations. Useful reference for how real-world tool wiring works inside a harness.
- OpenHands — The most architecturally complete open-source coding agent: Runtime/Sandbox isolation, EventStream message bus, and Agent Controller are a three-layer harness design worth studying for production deployments.
- Goose — Block's open-source, extensible AI agent donated to the Linux Foundation's Agentic AI Foundation in April 2026. Its MCP-native architecture treats every capability as an MCP server, making it a practical reference for building vendor-neutral, extensible harnesses where tool integration is the primary extension mechanism rather than framework-specific plugins.
- browser-use — Minimal browser-automation agent harness with clean separation of tool registration, DOM state injection, action loop, and error recovery. Small codebase, clear structure — the best "minimal viable harness" reference for understanding core loop mechanics.
Adjacent Collections
- EvoMap/awesome-agent-evolution — April 2026 curated list covering agent evolution, memory systems, multi-agent architectures, and self-improvement. Complements this list with a forward-looking lens on the next generation of agent capabilities — where harnesses must adapt to agents that modify their own scaffolding over time.
- Picrew/awesome-agent-harness — Implementation-first curated list (April 2026) with 150 entries, 84% GitHub projects, organized into 9 categories from harness architecture to sandboxing. The featured blogs section and catalog-style organization make it a strong complementary reference to this list's article-centric approach.
- jiji262/awesome-harness-engineering — Focuses on platform delivery governance, IDP, GitOps, and AI-native engineering. Overlaps with this list on the platform engineering side; more Harness-the-company oriented.
- VoltAgent/awesome-ai-agent-papers — Curated collection of 363+ arXiv papers from 2026 organized into five harness-relevant categories: Multi-Agent (51), Memory & RAG (56), Eval & Observability (79), Agent Tooling (95), AI Agent Security (82). Weekly updates make it the best single source for tracking research that will shape harness design decisions in 2026.
- bradAGI/awesome-cli-coding-agents — Catalog of 80+ terminal-native AI coding agents (open-source and proprietary) plus the harnesses that orchestrate, sandbox, and extend them: session managers, parallel runners, autonomous loop infrastructure, and credential vaults. The most comprehensive reference for the CLI agent layer that most harness infrastructure is designed to host.
Security, Sandbox & Permissions
- Beyond Permission Prompts — The authoritative resource on moving from prompt-level permission grants to structured authorization in the harness.
- How we contain Claude across products — Anthropic's May 2026 cross-product containment write-up: why environmental isolation must be the primary boundary, how model-layer defenses alone miss ~17% of overeager actions, and concrete sandbox architectures for chat, terminal, and autonomous workspace products. The exfiltration-through-allowlist case study is a stark reminder that the weakest link is often custom harness plumbing, not the sandbox itself.
- Model Context Protocol — Authorization — MCP's specification for OAuth-based authorization flows when agents access external services.
- AI Harness Scorecard — Scores repositories on AI harness safeguards. Useful checklist for auditing your own harness's security posture.
- E2B — Firecracker microVM sandboxes purpose-built for agent tool loops: ~150ms cold start, Python/JS SDKs, open source. The clearest reference implementation of "code execution as a harness primitive" rather than a CI system bolted on.
- tldrsec/prompt-injection-defenses — The most complete catalog of practical prompt injection defenses (input validation, tool output sanitization, canary tokens, etc.). Functions as a design checklist for hardening trust boundaries in any agent harness.
Evals & Verification
- Demystifying Evals for AI Agents — Anthropic's comprehensive guide to agent evaluation: trajectory evals, outcome evals, and how to build eval harnesses that are themselves reliable.
- DeepEval — The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates.
- Claw-Eval — 300 human-verified tasks across 9 categories evaluating LLM-as-agent performance on completion, safety, and robustness with a Pass^3 methodology that requires success across three independent trials. Referenced by Meta, Kimi, Qwen, and Tencent as a trustworthy benchmark for general agentic capabilities — the most rigorous community-verified eval harness published in 2026.
- SWE-bench — The canonical benchmark for coding agents. Essential reference for understanding what "verified working" means for harness outputs.
- Inspect AI — UK AI Security Institute's eval framework with native support for evaluating external agents (Claude Code, Codex CLI) as black-box targets, plus built-in bash/python/web browsing tools. Built for safety-grade rigor; the right foundation for harness-level eval infrastructure.
- Quantifying Infrastructure Noise in Agentic Coding Evals — Anthropic's empirical study showing container resource configuration alone produces 6+ percentage point benchmark swings — often exceeding model-to-model gaps. The 3x threshold finding is the key practical result: scores are stable up to 3x specified resources, but above that agents shift strategy entirely (lean tools vs. heavy dependencies), meaning tight and generous resource limits measure fundamentally different behaviors. Essential reading before interpreting any agentic eval.
Templates
Reusable starting points for harness artifacts. Copy and adapt.
Production Infrastructure & Operations
- Claude Managed Agents: Self-Hosted Sandboxes and MCP Tunnels — Anthropic's May 2026 enterprise deployment pattern keeps the agent orchestration loop on Anthropic's infrastructure while moving tool execution into customer-controlled sandboxes; combined with MCP tunnels for private-network tool access, it's the reference architecture for data-residency-conscious production harnesses.
- AgentCgroup: Understanding and Controlling OS Resources of AI Agents — February 2026 empirical study of sandboxed coding-agent workloads finding that OS-level execution accounts for 56–74% of end-to-end latency and memory is the real concurrency bottleneck (15.4× peak-to-average spikes driven by tool calls). Proposes an intent-driven eBPF controller aligned with tool-call boundaries — essential for anyone running multi-tenant agent sandboxes where coarse container limits waste resources or let agents starve each other.
- AI Agent Scaling Gap: Pilot to Production (March 2026) — Analysis showing 72% of Global 2000 companies operate agents beyond experimental phases, but only 14% successfully scaled organization-wide. Scaling success correlates strongly with operations infrastructure (monitoring, evaluation harnesses, incident response) rather than technology choices. Documents the shift from engineering-focused to operations-focused agent deployment.
- builderz-labs/mission-control — Self-hosted orchestration dashboard for agent task dispatch, multi-agent workflow coordination, and spend monitoring across gateways. The zero-external-dependency design (SQLite, single pnpm start) makes it the most practical open-source control plane for teams that need governance and cost visibility without building infrastructure from scratch.
- 5 Production Scaling Challenges for Agentic AI in 2026 — Data infrastructure prioritized before deployment; successful scalers appoint AI operations function pre-expansion; multi-agent distributed systems with load balancing and auto-scaling. Essential reading for understanding infrastructure prerequisites for agent deployment at scale.
Related Awesome Lists
Lists that cover adjacent territory — overlapping but not identical scope.
- Awesome Context Engineering — Comprehensive survey on context engineering: prompt engineering, RAG, context window management, production AI systems.
- awesome-claude-code — Curated resources, tools, and workflows specifically for Claude Code users.
- awesome-mcp-servers — Comprehensive list of MCP servers for extending agents with external capabilities.
- awesome-ai-agents — Curated list of AI agents and agent frameworks, organized by use case. Useful for surveying the landscape of what harnesses are being built around.
- awesome-llm-apps — Collection of production LLM applications with source code across RAG, multi-agent, and tool-use patterns. Good reference for how harness primitives combine in real applications.
- ICLR 2026 MemAgents Workshop — Interdisciplinary workshop (April 27, Rio de Janeiro) covering episodic/semantic memory, knowledge graphs, vector databases, retrieval pipelines, temporal credit assignment, and context management for agentic systems. The canonical venue for memory architecture research and standards; accepts full papers (9pg), short papers (4pg), tiny papers (2pg).
- Awesome Code as Agent Harness Papers — Curated companion list to the survey, organizing research on code-centric agentic systems across interface, mechanisms, and scaling layers. A focused research map for anyone building harnesses where code is the executable scaffold rather than just the output.
Contributing
See CONTRIBUTING.md.
What belongs here: Resources that address a specific harness engineering problem (context, tools, planning, permissions, memory, verification, sandboxing). Each addition should include a 1–2 sentence note explaining why it's worth including — this is an opinionated list, not a directory.
What doesn't belong here: General AI/ML papers, model benchmarks unrelated to agent harnesses, tutorials on using specific models, product marketing.
License
CC0 — public domain dedication.
Acknowledgments
Thanks to linux.do — a vibrant tech community where many harness engineering ideas were discussed and refined.
