Every agent team eventually runs into the same trade-off. Tighten the guardrails and the agent stalls halfway through a refund, blinking at a blocked tool call like an office printer. Loosen them, and the task completes, along with a policy violation your compliance team finds during the next audit.
A recent paper accepted as an oral at NeurIPS 2026 says that trade-off is optional. Its framework, Soteria, combines formally verified planning with runtime enforcement. Across GPT-5.4-mini, Gemini 3 Flash, Claude Haiku 4.5, and Claude Opus 4.6, it reached 100% conformance with zero specification violations.
Task success averaged 1.7 times the best baseline.

Why LLM agent guardrails keep failing in production
Most agent safety today follows one of two patterns. LLM-as-judge guardrails, such as NVIDIA’s NeMo Guardrails, ask a second model to approve each action, which amounts to hiring a second intern to supervise the first.
Specification-based monitors are stricter, checking preconditions and trace invariants before a tool fires.
Both have the same blind spot: They react one action at a time, so a blocked call leaves the agent mid-workflow with a half-written state and only guesswork to find a way back. Blocking prevents the violation and strands the agent at the same moment.
That gap grows as agents take on actions that are hard to undo. Our breakdown of the six mistakes AI leaders keep making with agentic deployments traced most incidents to enforcement bolted on after launch. Soteria moves enforcement ahead of the very first tool call.

How Soteria verifies an agent’s plan before it acts
The plan then goes to Dafny, an automated verifier with all the warmth of a tax auditor, which checks it against formalized policy.
Four mechanisms do the heavy lifting:
- Tool contract checks: every call must satisfy its precondition, so a refund tool only receives an amount derived from a real reservation lookup.
- Progress checks: the verifier confirms every state stays reachable and some guard can always fire, eliminating dead ends where the agent freezes mid-task.
- Policy compliance: each rule becomes a first-order formula over the execution trace, so “check flown status before cancelling” holds on every possible path.
- SpecMonitor at runtime: a monitor intercepts each proposed tool call, blocks violations, and gives the executor the verified plan as a recovery map.
In the paper’s ablation, this cut the median plan size to 8 states, down from 11, and raised the number of fully verified plans to 65, up from 38.
Fabricated tool arguments deserve a special mention. The paper’s retail example describes an agent processing a return with invented order data. That is a close cousin of classic AI hallucinations, except this one has database write access.
The agent safety results that should change your roadmap
Across 207 tasks on τ²-bench and Toolathlon, every baseline leaked violations. The best baseline conformance topped out at 93%. Claude Opus 4.6 running with zero guardrails still racked up 11 violations on the airline benchmark.
With Soteria, Claude Opus 4.6 completed 88% of airline tasks and 90.4% of retail tasks, compared with 56% and 46.5% for the strongest baselines. Smaller models gained even more relative ground: Claude Haiku 4.5 hit 70% on airline tasks, against 30% for its best baseline.
That echoes how the right tooling helps small AI models outperform the giants.
The paper only counts a task as a success when it completes with zero violations, and that definition exposes a pattern most dashboards hide. On Toolathlon, Opus completed 30.2% of tasks with zero guardrails in place. Once violations were subtracted, more than half of those were lost, leaving 11.6% genuine success.

Where formal verification for AI agents gets expensive
Soteria carries a price tag. On conversational benchmarks, it generates a fresh plan at every dialogue turn. That pushed per-task cost to $0.796 with Opus on airline tasks, roughly 2.4 times the baselines. Anyone wrestling with multi-turn reasoning will recognize that bill.
Single-turn work flips the math. On Toolathlon, structured plans cut wasted steps, and Soteria sometimes came in cheaper than the guardrails it beat. Writing the specifications also stayed manageable: a few hours per benchmark, producing between 24 and 41 formulas, drafted by an LLM and validated by people.
That human validation step is a good job description for the emerging AI architect. Somebody has to read the refund policy clause by clause, because Dafny will happily prove whatever you encoded wrong.
What agent builders should do next
Our position is that pre-execution verification belongs in every agent workflow that moves money or writes to customer records. A few extra cents per task costs far less than an incident review.
Practical steps:
- Formalize your highest-risk policies first: start with the handful of rules that would trigger an incident report, then expand coverage as the specs prove their worth.
- Build a sub-plan library: encode standard procedures once, verify them, and let planners compose them. It is the same reuse discipline that keeps mission-critical ML systems stable.
- Measure conformance-gated success: report task completion and violations together, alongside probes like canary tools for agent tool selection.
- Use strict mode for high-stakes flows: Soteria’s plan-enforcement mode permits only guard-satisfying transitions. It is a sensible default for payments and for shadow AI agents moving into sanctioned workflows.

Proof beats vibes for agent reliability
Agent evaluation has spent a long stretch in benchmark theater, where leaderboard scores stand in for trust. Soteria offers something sturdier: a guarantee you can audit.
It fits the wider shift toward demonstrating reliability as the entry ticket for enterprise AI, and slots neatly into an LLMOps stack built for the agentic era.
Formal methods will earn their place case by case, and yes, the honest answer still starts with “it depends.” For any workflow where a wrong call costs real money, a verified plan deserves a spot on the list of things every AI engineer should ship.
Bring the verification debate to London
Soteria’s central question, whether an agent finished the job and followed the rules, gets a full day on stage at the Agentic AI Summit London on December 1, 2026, at the InterContinental London, The O2.
Engineers and executives running agents in banking and healthcare will be in the room comparing notes on what actually holds up in production.
- A panel built around conformance-gated success: “Your agent completed the task. But did it do it right?” brings together practitioners from Citi and Meta to unpack the gap between finishing a task and behaving correctly.
- Production failure stories, shared candidly: speakers from Barclays, BlackRock, Admiral Group, and Great Ormond Street Hospital dig into why agent systems break once real users arrive.
- Evals and observability in depth: a session on taming non-determinism in agent trajectories pairs with a roundtable on evals that catch failures before your users do.
- Governance for the C-suite: one pass also covers the co-located Chief AI Officer Summit, including a panel with NatWest Group and JP Morgan Chase on balancing autonomy, trust, and risk.
Bring your hardest agent failure story. Someone in the room will have a spec for it.
Secure your seat at world.aiacceleratorinstitute.com/location/agenticailondon



