Harness Engineering

Before engineering solutions, it helps to diagnose the disease. Research from Composio, OWASP, and production post-mortems across the industry point to five dominant failure modes.

Agents that cannot retrieve the right context at the right time make bad decisions — and make them confidently. Poor retrieval-augmented generation accounts for a significant share of production failures, typically manifesting as agents answering questions with outdated policies, irrelevant precedents, or hallucinated "facts" that sound plausible but are wrong.

In financial services, this means a KYC agent applying last quarter's sanctions list. Or a compliance agent citing a regulation that was superseded six months ago.

The fix is a context validation layer: retrieval freshness verification, relevance scoring with configurable thresholds, and fallback-to-human when retrieval confidence drops below acceptable levels.

The gap between what an API promises and what it delivers in production is the second-largest source of agent failures. Agents built against sandbox APIs discover that production APIs have different rate limits, different error formats, different authentication flows, and different data shapes.

Nearly two-thirds of all agent failures trace back to scope overreach and data quality issues — agents tasked with more than their underlying infrastructure can support.

What works: schema-validated tool interfaces with explicit error handling contracts, circuit breakers that degrade gracefully when downstream services fail, and integration testing that runs against production-equivalent environments. Not sandboxes.

This is the failure mode that keeps risk officers awake at night. OWASP classified it as ASI08 in their 2026 Top 10 for Agentic Applications.

In a multi-agent pipeline, the output of Agent A becomes the input for Agent B, whose output feeds Agent C. If Agent A hallucinates a data point, Agent B makes decisions based on that hallucination, and Agent C takes actions based on those decisions. By the time a human sees the final output, the original error has been laundered through multiple layers of plausible reasoning.

Three factors make this uniquely dangerous:

1. Semantic opacity — natural-language errors pass validation checks that would catch malformed JSON
2. Emergent behaviour — multiple agents create unintended outcomes that no single agent's logs explain
3. Temporal compounding — errors persist in agent memory and contaminate future operations

Real-world impacts already documented include auto-remediation agents suppressing genuine alerts, database agents acting on hallucinated instructions, and healthcare agents propagating incorrect treatment protocols to downstream systems.

The fix here is structural: inter-agent validation gates, output verification against ground-truth data sources, confidence-scored handoffs with automatic escalation, and immutable trace logging that lets you reconstruct the exact propagation chain post-incident.

Agents that poll APIs instead of reacting to events waste 95% of their API calls, burn through rate-limit quotas, and never achieve real-time responsiveness. In financial services, where market data and transaction states change in milliseconds, polling-based agents are architecturally incapable of meeting performance requirements.

Event-driven architectures with webhook-based triggers solve this. Message queues for asynchronous processing, and reactive agent loops that respond to state changes rather than periodically checking for them.

The firms in the successful 6% share a common architectural pattern. Their harness is not a single tool. It is a layered stack, each layer addressing a specific class of production failure.

Every agent interaction begins with input validation. Not just format checking — semantic validation that ensures the agent has the right context, the right permissions, and the right scope for the task.

In practice this means request classification (is this within the agent's defined scope?), context freshness verification (is the retrieved data current?), permission verification, and data completeness checks. If the agent does not have enough information to act, it should say so rather than guessing.

This is where harness engineering diverges most sharply from traditional software engineering. Constraints are not defensive coding. They are the primary mechanism for controlling agent behaviour.

Five concrete budget guardrails, enforced at the infrastructure level:

1. Loop and step limits — prevent infinite reasoning chains
2. Tool-call caps — prevent runaway API consumption
3. Per-run token budgets — prevent cost explosions
4. Wall-clock timeouts — prevent hung agents blocking resources
5. Per-tenant budgets with anomaly alerts — prevent one client's workload affecting others

The key metric shift here: from tokens consumed to Cost-per-Accepted-Outcome (CAPO). Measuring cost against business value delivered, not computational resources used.

Production agents need closed-loop systems that detect errors and correct them before they reach end users.

This includes reflection middleware (the agent evaluates its own output before delivery), verification middleware (output checked against known constraints and business rules), loop detection (identifies when the agent is cycling without progress), and progressive escalation (automatic handoff to human review when confidence drops or corrections accumulate).

LangChain's composable middleware pattern illustrates the architecture: Agent Request → LocalContextMiddleware → LoopDetectionMiddleware → ReasoningSandwichMiddleware → PreCompletionChecklistMiddleware → Agent Response. Each layer adds a specific reliability guarantee.

AI agent observability has moved from developer convenience to mission-critical infrastructure. 89% of organisations have implemented observability for their agents, with quality issues emerging as the primary production barrier at 32%.

Agent observability differs from traditional software monitoring because agents operate non-deterministically with multi-step reasoning chains spanning LLM calls, tool usage, retrieval systems, and complex decision trees.

Production-grade agent observability requires trace-level logging of every LLM call, tool invocation, and decision point. It requires semantic evaluation — not just "did it respond" but "was the response correct and appropriate." It requires real-time cost tracking with automatic alerts and circuit breakers, per-step latency monitoring, and drift detection to identify when agent behaviour shifts from established baselines.

Gartner predicts that by 2028, 60% of software engineering teams will use AI evaluation and observability platforms — up from 18% in 2025.

The shift requires three changes.

Standard AI evaluation uses pass@k: if one out of k trials succeeds, the test passes. Production harness engineering demands pass^k: all k trials must succeed. The difference is profound. pass@k tells you the agent can succeed. pass^k tells you it reliably succeeds.

Five measurable dimensions for agent validation:

1. Consistency — same input produces acceptably similar outputs
2. Robustness — the agent handles edge cases and adversarial inputs without failing silently
3. Predictability — behaviour stays within defined bounds
4. Safety — no harmful outputs, no policy violations
5. Cost stability — no runaway spend across varied workloads

When you can measure governance, you can improve it. Five metrics that matter:

• Policy enforcement rate — percentage of decisions that passed through all required governance checkpoints
• Escalation accuracy — percentage of escalated decisions that genuinely required human review
• Audit trail completeness — percentage of agent actions with full provenance chains
• Time-to-intervention — how quickly humans can identify and halt problematic agent behaviour
• Compliance coverage — percentage of regulatory requirements mapped to running harness controls