Harness Engineering for AI Coding Agents

The term gained traction in early 2026 after Mitchell Hashimoto (co-founder of HashiCorp) articulated a principle that many engineering teams were already discovering the hard way: whenever an AI agent makes a mistake, the fix should be a change to the system, not a change to the prompt. OpenAI formalized the concept shortly after, describing how they built an entire internal product with zero lines of manually written code by investing heavily in the scaffolding around their Codex agents rather than in the prompts that drove them.

The Thoughtworks Technology Radar (Volume 34, April 2026) identified harness engineering as one of the defining macro trends in the industry. Their assessment: the field is moving from experimentation to a desire for repeatability and stability. Agent reliability has shifted from one concern among many to one of the most critical issues facing engineering organizations.

Here is the core problem. Telling an agent "follow our coding standards" in a prompt is like telling a new hire to "write good code." It works some of the time. But LLM compliance with instructions is probabilistic, not deterministic. The agent might follow your architecture conventions perfectly on Monday and introduce a direct database call that bypasses your ORM layer on Tuesday. Prompts don't enforce; they suggest.

Harness engineering adds the enforcement layer. AGENTS.md files encode architectural boundaries in a format agents can reference. CI gates block code that violates constraints before it reaches main. Plan-Execute-Verify loops force agents through structured checkpoints. And observability dashboards track whether the harness is actually working or whether new failure patterns are emerging. Each layer reinforces the others. The constraint layer reduces failure volume, the verification layer catches what slips through, and the observability layer surfaces patterns that both layers missed.

Plan

The agent reads the specification, loads project context from AGENTS.md and rules files, decomposes the task into subtasks, checks constraints, and proposes an implementation approach. For high-risk changes, a human approval gate sits between plan and execution. The agent cannot proceed until the approach is signed off.

Execute

The agent generates code and tests within a sandboxed environment. Execution is bounded by the constraints defined in the plan phase. Changes are committed incrementally, not as one massive diff. If the agent attempts to modify a protected module or introduce a disallowed dependency, the constraint layer blocks the action before it reaches the codebase.

Verify

The full test suite runs at every level: unit, integration, end-to-end. Static analysis checks for code quality and security issues. Architecture compliance verifiers confirm the change respects module boundaries. Mutation testing validates that the new tests actually catch real bugs, not just exercise code paths. Pass/fail feedback goes back to the agent, which can retry with the specific failure context.

The AGENTS.md specification, released in August 2025 as an open standard, gives agents structured project context. But context alone is not a harness. Most teams we work with have some form of AGENTS.md or CLAUDE.md file when we arrive. The files typically describe the tech stack, list some conventions, and maybe include a few "don't do this" instructions.

That is a starting point, not a constraint system. A well-engineered AGENTS.md includes architectural boundaries that map to enforceable CI checks, testing requirements tied to specific verification gates, security constraints that reference actual scanning rules, and dependency policies that match your package management configuration. The file becomes a contract between the human team and the agents, not a suggestion list.

We build AGENTS.md files that function as executable specifications. Every constraint in the file maps to a CI check that enforces it. Every testing requirement maps to a verification gate. When the file says "all database queries go through the ORM layer," there is a static analysis rule that catches direct SQL calls. When it says "integration tests required for API routes," the CI pipeline blocks merges without them.

Phase 1: Agent Audit (Weeks 1-2)

We map your current agent usage across teams and repositories. Which tools are developers using? What percentage of recent commits involve AI assistance? Where are the failure clusters? We measure baseline metrics: task resolution rate, rework rate, architecture drift. The output is a prioritized inventory of harness requirements and a practical implementation plan.

Phase 2: Harness Design (Weeks 3-5)

Based on the audit, we design the constraint architecture. Which boundaries need enforcement? Where do verification gates go in your CI pipeline? What level of human oversight is appropriate for different risk levels? How should multi-agent workflows be coordinated? We make these decisions collaboratively with your team. Harness engineering does not work if it is imposed from outside.

Phase 3: Build and Integrate (Weeks 5-10)

The implementation phase. We author AGENTS.md and rules files for each repository, deploy PEV loop infrastructure, configure CI gates, set up agent sandboxing, wire in observability dashboards, and calibrate constraint thresholds. Every component is tested against your actual agent workflows, not theoretical scenarios. Thresholds are tuned to avoid over-constraining: a harness that blocks too much valid work is worse than no harness at all.

Phase 4: Handoff and Training (Weeks 10-14)

Documentation, runbooks, spec-writing workshops, and hands-on training for maintaining the harness as your codebase and agent tools evolve. We train your team to update constraints, adjust verification gates, and interpret the observability data. The goal is your team running the harness independently. We offer ongoing support as an option, but the default assumption is full ownership transfer.

The Situation

A Series B SaaS platform with about 70 engineers came to us in early 2026 after hitting a wall with their AI coding tool adoption. Their team had enthusiastically adopted Claude Code and Cursor Agent six months earlier. Individual developer throughput had increased by roughly 40 percent based on their internal tracking. Leadership was pleased with the numbers.

Then three things happened in the same month. A production incident traced back to an AI-generated database migration that bypassed their ORM layer and introduced a data integrity issue affecting 12,000 customer records. A security audit found credential handling inconsistencies across 8 of their 14 repositories, all in AI-generated modules. And their lead architect realized that their monorepo had developed two competing API patterns because different agents had introduced different conventions in different services, and nobody caught the drift until it caused integration failures.

They tried fixing these problems with better prompts and more detailed system instructions. It helped for about two weeks. Then new instances of the same patterns appeared in different parts of the codebase. The engineering VP told us: "We cannot go back to the old speed, but we cannot keep shipping at this defect rate either."

What We Built

We deployed a full harness engineering implementation over 12 weeks with a five-person team: two platform engineers, two DevOps specialists, and a harness architect leading the engagement.

The critical decisions were:

• AGENTS.md files per repository with enforceable architectural boundaries. The ORM-only constraint for database access was backed by a Semgrep rule that blocked direct SQL in CI. The duplicate API pattern problem was addressed with a shared API style guide that agents had to reference.
• PEV loops in GitHub Actions that required agent-generated PRs to pass plan validation, automated testing at three levels, and architecture compliance checks before merge was allowed.
• A multi-agent coordination layer that assigned Claude Code to complex refactoring tasks and Cursor to routine feature work, with task decomposition to prevent merge conflicts between concurrent agent sessions.
• Observability dashboards tracking agent success rate, rework rate, and drift detection across all 14 repositories, integrated with their existing Datadog setup.

Six months after deployment, the team maintained the 40 percent throughput gain from AI tools while reducing the defect rate below pre-agent levels. The engineering VP's summary: "We went from fighting the agents to working with them." Want to see more of our work? Visit our case studies page.

Most companies approach harness engineering as a purely technical challenge. Build the CI gates, write the rules files, deploy the dashboards. Done. But the hardest failure mode we see is not technical. It is cognitive debt: the growing gap between what your codebase does and what your team understands about it.

When agents generate 60 or 70 percent of your code, developers lose touch with implementation details they used to know intimately. A senior engineer who used to understand every quirk of the authentication module now reviews agent-generated changes to it without the deep context they once had. The code works, the tests pass, but the human understanding erodes. Six months later, when something breaks in a way the tests did not anticipate, nobody has the mental model to debug it quickly.

This is why our engagements always include the team enablement component. We help engineering managers identify which parts of the codebase require deep human understanding (security-critical paths, data integrity logic, customer-facing payment flows) and which parts can safely become "agent-maintained territory" with lighter human oversight. The goal is not maximizing agent autonomy. It is finding the right balance between velocity and understanding for each team.

Outsourcing Is a Good Fit When

• You have no platform engineers with agent reliability experience on staff (most teams do not, since the discipline is less than a year old)
• Agent-generated code is already causing production issues and you need harnesses operational in weeks, not quarters
• Your team uses multiple AI coding tools and you need harnesses that work across all of them
• You want the harness built once, tuned to your codebase, and handed off to your internal team to maintain
• You have more than 30 engineers using agents and the coordination overhead is becoming unmanageable

Building In-House Makes More Sense When

• You already have a strong platform engineering team that just needs guidance and a reference architecture
• Your team is small enough (under 15 engineers) that informal review processes still catch most issues
• You use a single AI coding tool and your constraint needs are straightforward
• You have 6 to 9 months to iterate on the harness without urgent production pressure

Task Resolution Rate measures the percentage of tasks an agent resolves correctly, verified by automated tests. Industry benchmarks from DORA research suggest top agents achieve 65 to 77 percent resolution rates in well-harnessed environments. Without harnesses, most teams see rates between 25 and 40 percent.

Pass@1 Rate tracks whether the agent gets it right on the first attempt without retries. This matters because retries consume compute budget and developer attention. A well-tuned harness pushes pass@1 above 60 percent for routine tasks.

Rework Rate measures what percentage of agent-generated PRs require human intervention after the first verification pass. Before harness deployment, rework rates of 50 to 60 percent are common. After deployment, we target below 15 percent.

Architecture Drift tracks how often agent-generated code violates established architectural patterns. This is the metric that surfaces the slow degradation problems that individual PR reviews miss. We integrate drift detection into your existing observability stack.

AI
Development Outsourcing

AI Code
Security Services

AI-Powered
Software Testing

AI Agents
Development Outsourcing

MCP
Development Outsourcing

Platform
Engineering Outsourcing