Anatomy of a SOTA Agentic SOP-Execution Engine

There is a phrase gaining traction in the AI engineering community: harness engineering. Prompt engineering concerned itself with crafting the right input to an LLM. Harness engineering concerns itself with crafting the right execution environment around it — shifting emphasis from “what you say to the model” to “what you let the model do, when, and how you constrain it.”

Proceda, run against Amazon’s SOP-Bench — a benchmark with 2,411 tasks across 14 business domains — achieved state-of-the-art on 4 of 10 runnable domains, with 8 of 10 on SOP-consistent tasks. The models used were Gemini 2.5 Flash and Gemini 3 Flash. The baselines they beat were Claude 4 Opus, Claude 4.1 Opus, and Claude 4 Sonnet.

This post is a technical design analysis of how. It is a companion to the results report (which covers the numbers) and the 48-hour journey post (which covers the story). This piece covers the architecture.

The Thesis

Procedural tasks have defined steps, tool dependencies, and deterministic outcomes — Patient Intake, KYC compliance, Dangerous Goods classification, Content Moderation, Order Fulfillment. These are not open-ended reasoning problems. They are procedures: sequences of steps with tool calls, decision gates, and branching logic that must be executed faithfully.

SOP-Bench tests exactly this. Each domain provides a multi-step Standard Operating Procedure, a set of mock tools, and ground truth labels. The question: can an AI agent follow the procedure, call the right tools, and produce the correct output?

The baselines in the SOP-Bench paper use two standard agent architectures:

• Function Calling (FC): The full SOP is placed in the system prompt alongside all tool schemas. The model determines the sequence autonomously.
• ReAct: The same setup, augmented with a thought-action-observation loop.

Both treat the SOP as prompt context. Proceda treats it as a state machine.

How Baselines and Proceda Differ

This represents a category difference, not a degree difference. The baseline architectures require the model to function as both planner and executor. Proceda separates those concerns: the SOP defines the plan, the harness manages the execution lifecycle, and the model handles one step at a time.

Seven Design Decisions That Enable Cheaper-Model SOTA

Each decision below is connected to a specific benchmark outcome. The pattern is consistent: the harness absorbs complexity that would otherwise fall on the model.

Decision 1The SOP as State Machine

A SKILL.md file is parsed into a sequence of SkillStep objects. The executor walks through them with a simple loop:

while current_step <= total_steps:
    prompt LLM with step instructions
    run LLM loop until complete_step() is called
    handle approval gates if needed
    advance to next step

The model never receives a prompt like “you are on step 3 of 12, here is what you did in steps 1 and 2, now figure out what to do next.” Instead, it receives: “Execute Step 3: Validate insurance. Here are the instructions. Here are the tools. Call complete_step when done.”

This design decision is not purely technical. Stepwise execution also matches the mental model of the business users who author and oversee SOPs. Operations teams think in steps — they want to see which step the agent is on, review progress at each stage, and designate specific steps as requiring human sign-off before proceeding. A monolithic prompt that produces a single final answer offers no such visibility. The state machine architecture serves both the LLM (by reducing cognitive load per call) and the human operator (by providing the step-level observability and control they expect from any process automation system).

Decision 2Control Tools as Explicit Progress Signals

Two “control tools” are always injected into every LLM call alongside the domain’s actual tools:

• complete_step(summary) — the LLM signals that the current step is done
• request_clarification(question, options) — the LLM asks the human for input

The model must call complete_step to advance. There is no implicit step detection — no pattern matching on the model’s text output to infer completion. The model makes an explicit, structured declaration.

This prevents the model from silently skipping steps, rambling past step boundaries, or confusing “thinking about the step” with “completing the step.”

The harness includes a two-tier fallback for models that produce text without calling tools:

1. Soft nudge (every N text-only responses): “You seem stuck. Call complete_step if done, or use a tool to make progress.”
2. Hard cap (5× the nudge threshold): Force-complete the step and move on.

Decision 3Context Trimming (The Forgetting Strategy)

The context manager applies aggressive token budgeting:

1. Always preserve system messages (the skill definition)
2. Always preserve “critical” messages (step prompts, clarification responses)
3. Fill remaining token budget with recent non-critical messages, newest first
4. If anything was trimmed, insert a notice

The step prompt for the current step is marked as critical at creation time, so it is never trimmed. Old tool results from previous steps are the first to go.

Decision 4Structural Approval Gates

SKILL.md supports two approval markers:

• [APPROVAL REQUIRED] — human reviews after step completion, before advancing
• [PRE-APPROVAL REQUIRED] — human must approve before step execution begins

These are parsed from the markdown by the parser (regex-based marker extraction) and enforced by the executor’s step loop. The LLM never decides whether to pause for approval. The harness makes that decision based on the SOP’s structural markers.

Decision 5output_fields and Structured Extraction

The SKILL.md frontmatter can declare output fields:

output_fields:
  - final_resolution_status
  - escalation_required

When present, the system prompt instructs the model to emit XML tags in its final complete_step summary:

<final_resolution_status>RESOLVED</final_resolution_status>
<escalation_required>NO</escalation_required>

The extractor then parses these deterministically — no fuzzy matching, no regex on prose, no ambiguity between “RESOLVED” and “the status should be resolved.”

Decision 6Guard Rails as Circuit Breakers

The executor enforces hard limits on every step:

The tool call circuit breaker merits attention. If a step exceeds 20 app tool calls, execution does not crash — it pauses and asks the human (or auto-approve in benchmarks) whether to retry (reset the counter), skip the step, or cancel the run.

Decision 7Event-Driven Observability

Every runtime transition emits a structured event: step started, tool called, tool completed, LLM usage, approval requested, status changed, and 20+ other types. These are written to a JSONL trace file for every task.

This may appear to be a nice-to-have. In practice, it was the mechanism that made every other improvement possible.

• The output_fields breakthrough came from trace analysis. The traces showed 100% ECR but 30% TSR on Customer Service — and revealed exactly where extraction was failing: the model’s answer was in the complete_step summary, but the extractor was picking up a field from an earlier tool result.
• The benchmark bug discoveries came from traces. When Proceda follows the SOP correctly but scores wrong, the trace proves it. This is how the analysis revealed that Traffic Spoofing’s CSV labels contradict its SOP for 39 tasks.
• The thinking model analysis came from traces. The traces revealed the model spending tokens on extended reasoning about straightforward arithmetic — reasoning that provided no benefit and sometimes introduced errors.

The Thinking Model Paradox

The most counterintuitive result from the benchmark run was the model comparison on Referral Abuse v2:

The thinking model — the one that spends extra compute on chain-of-thought reasoning — performed worst. The cheap non-thinking model beat it by 14 points.

The explanation: Referral Abuse requires calculating penalty scores from a table, comparing them, and selecting the right violation. The procedure is arithmetic and table lookups, fully specified by the SOP. The thinking model treated each task as a reasoning puzzle, sometimes second-guessing the SOP’s instructions or over-analyzing edge cases that were not edges.

Thinking models solve a different problem: they help when the task requires figuring out what to do. Structured SOP execution already specifies what to do. The model need only execute.

The Cost Curve Inversion

Gemini 2.5 Flash is roughly an order of magnitude cheaper per token than Claude 4 Opus. On Dangerous Goods, the cheaper model wins by 7.2 percentage points. On Aircraft Inspection, it ties (100% vs 99%). On Patient Intake, it trails by 3 points — but the baseline required Claude 4.1 Opus, the most expensive model in the comparison.

The pattern suggests a boundary. For domains with deterministic decision logic — even complex multi-step logic like Dangerous Goods classification — cheap models with structured execution match or beat expensive models with unstructured execution.

For domains requiring subjective judgment — like Know Your Business, where the SOP says to escalate on risk indicators but the ground truth follows unstated rules — stronger models retain an advantage. Proceda scored 42.2% on Know Your Business with Gemini 3.1 Pro, versus 58% for the Claude 4.5 Opus ReAct baseline.

Proceda is open source at github.com/vivekhaldar/proceda. Full benchmark results are in the SOP-Bench results report. The 48-hour benchmarking journey is in How We Achieved SOTA on SOP-Bench in 48 Hours. Built by Enchiridion Labs.