Terminology

Why SOPs Matter

Standard Operating Procedures are how enterprises encode critical business processes — from patient intake to KYC compliance to content moderation. Unlike informational knowledge bases (where benchmarks like SkillsBench measure factual retrieval), SOPs are procedural: they define a sequence of steps, tool calls, decision gates, and branching logic that must be executed faithfully.

Automating SOPs with AI is a key enterprise use case. But “dump the SOP into a prompt and hope for the best” doesn't work — as SOP-Bench demonstrates, even frontier models fail when the procedure is complex. The patient intake domain, rated the easiest by human experts, scores 0% TSR with both baseline agent architectures when using Claude 3.5 Sonnet v2 (the model shipped with the SOP-Bench evaluation code). The paper’s Table 5 shows that scaling to Claude 4.1 Opus achieves 100% — but that requires a frontier model on the simplest domain.

SOP-Bench is a welcome benchmark in this space. It provides 14 real-world business domains with mock tools, ground truth labels, and standardized evaluation. It tests what matters: can an AI agent reliably follow a multi-step procedure, call the right tools with the right parameters, and produce the correct output?

---

Results

Performance Summary

* SOP-consistent TSR excludes tasks where the benchmark's CSV ground truth contradicts the SOP's explicit rules. Baseline TSR is measured on all tasks, so the comparison favors the baseline. Baselines are best-across-all-models from SOP-Bench paper v2 Table 5.

Not Run (4 domains — benchmark bugs)

Visualizations

---

Cheaper Models, Better Results

One of the most striking findings is the model cost story. The SOP-Bench paper's best baselines require frontier models — Claude 4.1 Opus, Claude 4.5 Opus, Claude 4 Sonnet. Proceda beats these with much lighter models:

The only domain that required Gemini 3.1 Pro where the SOP explicitly asks the agent to exercise subjective judgment is Know Your Business — the SOP says “use your experience” to distinguish typos from fraud, and notes that “risk scores are not reliable.” Referral Abuse v2 also used 3.1 Pro, but primarily because it was the model that achieved SOTA during our model comparison experiments.

For domains with clear, formulaic decision logic — even complex ones like Dangerous Goods (274 tasks, weighted scoring with imputation) — cheap models suffice when execution is structured.

---

Benchmark Quality Contributions

During this evaluation, we identified and reported systematic issues in the SOP-Bench data. We frame these as contributions to the benchmark's quality — SOP-Bench is a valuable new benchmark, and these findings help improve it.

Domains with broken tools (4 domains, not runnable)

SOP/CSV labeling disagreements (4 domains)

These domains have functioning tools and were run successfully, but a portion of tasks have ground truth labels that contradict the SOP's explicit rules. An agent faithfully following the SOP gets penalized.

The SOP-Bench paper lists “implicit knowledge that humans learn but rarely document” as a benchmark challenge. However, each agent evaluates tasks in isolation with no access to ground truth labels — there is no mechanism to learn unstated rules from data patterns.

---

How Proceda Works

The Problem with Existing Approaches

The SOP-Bench paper evaluates two agent architectures: Function Calling (tool use in a single prompt) and ReAct (thought-action-observation loop). Both dump the entire SOP as raw text into a single prompt and say “follow this.”

This works for short procedures. It fails for complex ones. Patient intake — rated the easiest domain by human experts — requires a 6-tool dependency chain where the final tool needs outputs from all 5 previous tools as input parameters. With Claude 3.5 Sonnet v2, both baseline architectures score 0% TSR. Scaling to Claude 4.1 Opus achieves 100% — but requiring a frontier model for the simplest procedure highlights the brittleness of the unstructured approach.

Convert, Structure, Execute

Proceda treats SOPs as first-class executable artifacts, not prompt context.

1. Automated Conversion. proceda convert uses an LLM to transform unstructured SOP text into a structured SKILL.md file — a markdown document with YAML frontmatter and ### Step N: headings. The --tools flag passes tool schemas so the converter generates steps referencing exact tool names and parameter names. The --output-fields flag declares expected outputs so the final step emits structured XML tags. No hand-editing.
2. Structured Execution. The runtime executes each step sequentially. The LLM sees only the current step's instructions plus context from previous steps. Tool schemas are injected with exact parameter names. The LLM must call complete_step with a summary before advancing. Circuit breakers catch infinite loops.
3. Structured Output Extraction. When output_fields are declared, the system prompt instructs the LLM to emit <field_name>value</field_name> XML tags. The output extractor parses these deterministically — no fragile regex on free-form text.

Why This Enables SOTA with Cheaper Models

This is why Gemini 2.5 Flash beats Claude 4 Sonnet on Dangerous Goods: the procedure is formulaic — weighted scoring with imputation rules — and Proceda's step decomposition makes each step tractable. The model doesn't need to “be smart”; it needs to follow instructions and call the right tool with the right parameters.

For a deeper technical analysis of the seven design decisions that enable this — state machine execution, control tools, context trimming, approval gates, structured extraction, guard rails, and event-driven observability — see Anatomy of a SOTA Agentic SOP-Execution Engine.

Operational Features for Production SOPs

---

Conclusion

SOP-Bench demonstrates that faithfully executing complex procedures is a hard problem for AI agents. Proceda's structured approach — automated SOP conversion, step-by-step execution with tool integration, and human oversight — achieves state-of-the-art results on 4 domains outright and 8 of 10 on SOP-consistent tasks, often with significantly cheaper models than the published baselines.

The benchmark also has room to grow. We've filed 6 issues identifying broken tools and labeling inconsistencies across 7 domains, and hope these contributions help strengthen SOP-Bench as a standard for evaluating procedural AI.