I agent audit: 18 of 25 cognitive modules sat idle

Somewhere on GitHub right now, an agent framework lists "theory of mind" as a feature with all the confidence of a cereal box promising a balanced breakfast. A new paper posted to arXiv in late October 2026 checked what those labels actually compute.

For most of them, the answer was pure set dressing…

Cognitive architecture became a sales vocabulary

Agent builders love borrowed neuroscience. Global workspace, causal reasoning, active inference: drop those names into a module list and a local LLM agent suddenly sounds like a research lab in a box.

The audit paper names the core assumption head on: the presence of a module name gets treated as proof the mechanism exists.

Anyone who has sat through a vendor demo recognizes the pattern. Genuine causal reasoning in agents is hard, as recent work on agents struggling with why questions makes clear. A label on a Python class is considerably easier to ship.

18 of 25 modules hit a dead end

The researchers audited an LLM-based agent system advertising 25 cognitive mechanisms. In 18 of them, the computed value stopped short of every production output. That leaves 72% of the advertised cognition performing for an empty room, a hallucinated feature list that would make most AI hallucinations blush.

Rigor sets this paper apart from a snarky blog post. The team pre-registered falsifiable predictions for 11 modules, then ran 12 interventions against 61 frozen criteria.

Fifty-three held up and five were falsified outright, a refreshingly honest tally in an industry fond of benchmark theater. A replication on the IIT-phi core of a second, independent system used its own pre-committed falsification criteria.

Meet the idle-wiring family

The audit catalogs new idle-wiring shapes that dead-code and code-smell research had yet to name. The first sits wired into the agent graph and waits forever for a call. The second gets called, burns compute, and leaves the downstream state exactly as it found it.

The third runs on every request while ignoring what the request actually says, the software equivalent of a horoscope.

All of them slip past static analyzers because the code looks alive. Imports resolve, functions fire, logs fill up, and linters stay cheerful. The output simply routes around the module, like a meeting attendee whose comments somehow fail to reach the minutes.

That content-blind shape is a close relative of the failures exposed by canary tools research earlier this year: agents that look busy while their judgment sits idle. Teams running multi-turn reasoning pipelines pay the steepest price, because each inert step compounds across turns.

The repair numbers deserve a second read

💡

Two repair experiments served as positive controls. Rewiring lifted planner task success from 0.028 to 1.000, and state-controller error dropped by 52%.

Sit with that first figure. The planner hit a perfect score once its output connected to something, meaning the capability existed all along behind a broken pipe. For anyone focused on operational stability in mission-critical ML systems, the lesson stings: reliability failures can hide inside features everyone assumed were live.

Reliability now has to be proven, as AI's new rule of demonstrating reliability argues. This audit offers one of the most concrete ways yet to prove it.

What AI engineers should check this sprint

The paper's protocol relies on line-number evidence and call-chain tracing, and the authors frame it as reusable. Here is a practical version for your own stack, and a strong candidate for the list of things every AI engineer should have shipped:

Trace every named module to an output. Follow each return value along the call chain until it reaches a response or a state write. A dead end counts as a finding.
Ablate and compare. Stub the module out and rerun your eval set. Identical results mean the module is decorative, however impressive its name.
Pre-register expectations. Write down what each component should change before testing it, so the results have room to surprise you.
Probe content sensitivity. Feed a module varied inputs and confirm its outputs vary too. Static answers to shifting questions point to content-blind code.

Wire these checks into your LLMOps pipeline so audits run on every release, rather than once before a funding round.

A procurement problem dressed as a research finding

💡

For leaders, this is agent washing proven at the source-code level. A mid-2025 analyst estimate put genuine agentic capability at roughly 130 vendors out of the thousands marketing it.

This audit hands buyers a method to sort the rest, and it adds a seventh entry to the mistakes AI leaders keep making with agentic deployments.

Chief AI officers, including the most influential CAIOs in Silicon Valley, should bring these questions to every agent vendor review:

Which modules can we switch off during a trial? A vendor confident in its architecture will welcome ablation tests on your data.
Where does each module's output land? Ask for the call path linking "causal reasoning" with the final action.
What did your internal ablations show? Before-and-after numbers per module tell you more than a headline benchmark.
Who reviews the code behind the claims? Independent audit rights belong in any contract for agents touching production systems.
How do named reasoning modules shape escalation? A real uncertainty module should change when the agent hands work to a person.
Which modules appear in the audit trail? Components missing from logs deserve hard scrutiny.

Product teams share the responsibility. The applied ML product manager writing the feature list should be able to point to the code path behind every line on it.

Names are claims, so audit them

The paper's conclusion fits on a sticky note: module names and module computations are separate claims, and each deserves its own audit. Expect this to become standard due diligence within a year, especially as AI swarms multiply module counts and the AI architect takes ownership of system-level verification.

💡

A lean agent with every module fully wired will outperform a sprawling cognitive architecture where most of the brain is decoration. Ship the plumbing first. The poetry can wait.

Bring the audit mindset to New York

The Agentic AI Summit New York takes over Convene 360 Madison Avenue on June 3, 2027, bringing 500+ engineers and execs together during NY Tech Week. Here’s what a seat gets you:

Real-world agent evaluation, with sessions on testing autonomous agents under live conditions instead of polished demos.
Debuggable by design, covering how teams build agent workflows transparent enough to trace the moment something breaks.
Benchmarks against live architectures, spanning evaluation and continuous fine-tuning, so you know what your own stack should measure up to.
A room full of the right peers, with 350+ companies represented and 60% of attendees in senior management, so builders and buyers compare notes in one place.

Bring the checklist from this article and leave with a sharper one. Secure your pass at world.aiacceleratorinstitute.com/location/agenticainewyork.

Your AI agent's brain is 72% decoration. Here's the audit