> ## Content Index > Fetch the complete content index at: https://www.aiacceleratorinstitute.com/llms.txt > Use this file to discover other available public pages before exploring further. # Your AI agent's playbook is the bug - URL: https://www.aiacceleratorinstitute.com/your-ai-agents-playbook-is-the-bug/ - Published: 2026-11-06T11:57:58.000Z - Updated: 2026-11-06T11:57:58.000Z - Description: A new arXiv study pits simple LLM harnesses against five published root cause analysis agents. The plain setups won on accuracy and cost... - Author: Andrew Lovell - Tags: LLMs, Articles Somewhere in your stack sits an AI agent with a carefully staged investigation plan, a squad of role-based sub-agents, and an architecture diagram someone presented with genuine pride. A [**new arXiv paper**](https://arxiv.org/abs/2610.35767) suggests that diagram is the part holding the whole system back. ## **Root cause analysis became the favorite agent proving ground** Microservice root cause analysis (RCA) looks tailor-made for agentic AI. An alert fires, evidence sprawls across dashboards and log streams, and somebody has to name the faulty service before the on-call engineer finishes a third coffee. Research teams answered with increasingly elaborate LLM agents built on standard operating procedures and fixed investigation stages, often split across specialist role agents in the style of [**AI swarms**](https://www.aiacceleratorinstitute.com/ai-swarms-are-coming-heres-why-it-matters/). Each system typically arrives with its own dataset and a short list of baselines, so comparing them has drifted into benchmark theater. [AI observability: who's watching your AI agents?Agents reached production faster than the tools built to watch them. Here's what the new AI observability stack captures, what it costs, and where to start…![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/icon/AIAI-ICON-b40d67da-e71f-4f79-960b-da3f7892cfa1.png)AI Accelerator InstituteAndrew Lovell![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/thumbnail/AIAI_Website_Article_Images_Doodles--4--copy-6b17ce2b-a49d-4c33-a38e-cb8f88a1095a.png)](https://www.aiacceleratorinstitute.com/whos-watching-your-ai-agents-observability-just-had-its-breakout-year-2/) ## **Researchers stripped the scaffolding away** The study, posted in late October 2026, asks whether all that choreography beats handing a capable model the evidence and letting it investigate freely. The team built RCAtlas, a benchmark of 90 validated incidents drawn from eight public datasets, and ran it across six models at three reasoning-effort settings. **The contenders lined up like this:** - **A single LLM call** over a packaged bundle of incident evidence, about as simple as an agent setup gets. - **A SQL agent** with open query access to the telemetry, free to decide its own next move. - **The Codex coding agent** from [**OpenAI**](https://www.aiacceleratorinstitute.com/case-study-open-ai/), pointed at the incident and left to explore. - **Five published LLM RCA agents**, each hard-coding anywhere from a tenth of its model calls to all of them in the program. Statistical and causal methods rounded out the field as the non-LLM baseline. --- ## **The plain harnesses won on every model** The simple harnesses posted top-1 precision (PR@1) of 65% to 77% on three mid-tier models in the study. The published agents landed between 16% and 53%, while statistical and causal methods trailed at 6% to 39%. That ordering held on every model where both agent families ran. On a 2 am page, a gap that size separates rolling back the right service from rolling back your entire weekend. Cost tells the same story. Among runs that beat the best non-LLM method, every Pareto-optimal configuration for cost or time used a simple harness. Mid-tier models carrying these results also fit a broader pattern of [**smaller models outperforming the giants**](https://www.aiacceleratorinstitute.com/the-google-tool-helping-small-ai-models-outperform-the-giants/) once the setup suits them. [Self-building labs: AI agents that build their own labBehind every self-driving lab sits a human who taught a robot arm exactly where the vial rack lives, and who gets to teach it all over again the moment the experiment changes…![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/icon/AIAI-ICON-cf6ca419-bdd6-42cf-bea6-93ecb250fe9b.png)AI Accelerator InstituteAndrew Lovell![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/thumbnail/AIAI_Website_Article_Images_Doodles--4--10-6ed5f9ef-729b-4b16-8427-09d494063515.png)](https://www.aiacceleratorinstitute.com/self-building-labs-hand-ai-agents-the-keys-to-the-whole-lab/) ## **Scripted investigations look in the wrong place** The paper's case study pinpoints the mechanism. Hard-coded investigation paths skip the decisive evidence, because the prescription decides where to look before the incident reveals what matters. Production incidents live in the long tail. An SOP encodes the failures its authors remember, while a model with query access chases the failure actually in front of it. Diagnosis is a question at heart, a category of reasoning that [**agents handle better with the right memory and context**](https://www.aiacceleratorinstitute.com/ai-agents-struggle-with-why-questions-a-memory-based-fix/). The design lesson sits squarely in [**agent experience (AX)**](https://www.aiacceleratorinstitute.com/the-rise-of-agent-experience-ax/) territory: good tools and room to use them beat a script written for last quarter's outage. ## **Model tier, harness, and thinking time buy different things** The ablations double as a spending guide. A stronger model mainly improved[ **reliability**](https://www.aiacceleratorinstitute.com/ais-new-rule-demonstrating-reliability/), the consistency of correct answers across runs. A [**multi-turn**](https://www.aiacceleratorinstitute.com/is-multi-turn-reasoning-broken/) harness mainly improved aptitude, the ceiling of what the system could solve. Reasoning effort was the outlier. Turning it up moved accuracy by a few points at most while multiplying runtime by as much as seven. In agent terms, a maxed-out thinking budget is the equivalent of rereading the stack trace very slowly. For teams watching [**inference economics**](https://www.aiacceleratorinstitute.com/ais-new-era-train-once-infer-forever/), that trade looks like a seven-fold wait for a rounding error. [Toronto AI companies taking agentic AI to productionToronto pairs Hinton-era research depth with bank-sized AI budgets. Here is where the city's biggest AI players are building, and what they are shipping.![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/icon/AIAI-ICON-86c95782-b098-4f0a-9802-0bb6b7eb5a8f.png)AI Accelerator InstituteAndrew Lovell![](https://storage.ghost.io/c/26/b3/26b323cb-c378-4831-bc7d-27e29def746a/content/images/thumbnail/AIAI_Website_Article_Images_Doodles--4--9-0e552768-d980-446b-9d3d-d5e9a2e17c78.png)](https://www.aiacceleratorinstitute.com/toronto-is-where-ai-agents-go-to-production/) ## **What to change before your next agent sprint** The findings translate into a short list of moves for anyone building or buying agents for incident response: - **Benchmark a plain harness first.** Run a single call or a query-capable agent against your own incident history before writing a single SOP stage, the same measure-it-directly instinct behind [**canary testing**](https://www.aiacceleratorinstitute.com/the-canary-test-ai-agents-keep-failing/). - **Give the model data access, then step back.** The SQL agent's open query access let it follow the evidence wherever it led. - **Upgrade the model before the thinking budget.** Model tier buys reliability, while extra reasoning effort mostly buys latency. - **Demote SOPs to context.** Runbooks still hold hard-won institutional knowledge, so put them in the prompt as guidance and let your [**LLMOps pipeline**](https://www.aiacceleratorinstitute.com/llmops-optimizing-towards-enterprise-value-in-the-llm-agentic-era/) measure whether they help. --- ## **Simpler agents need sharper engineers** One benchmark of 90 incidents in one domain settles a narrow question, and how far it generalizes earns the eternal answer: it depends. The direction, however, is hard to argue with. Every layer of scaffolding encodes an assumption about where the answer lives, and capable models increasingly find it faster on their own. Anyone with the bitter lesson pinned above their desk has earned a small victory lap. The skill moves up the stack. The [**AI architect**](https://www.aiacceleratorinstitute.com/the-emergence-of-the-ai-architect-engineering-the-future-of-tech/) now designs data access, tools, and evaluation, then gets out of the model's way, a discipline that matters most for[ **mission-critical ML systems**](https://www.aiacceleratorinstitute.com/operational-stability-for-mission-critical-ml-systems/) sitting on the incident path. Leaders weighing build versus buy should ask every vendor for a plain-harness baseline, a habit that pairs well with avoiding the [**common agentic deployment mistakes**](https://www.aiacceleratorinstitute.com/6-mistakes-ai-leaders-keep-making-with-agentic-deployments/). Engineers can add "beat a simple harness" to the list of [**things every AI engineer should have shipped by now**](https://www.aiacceleratorinstitute.com/7-things-every-ai-engineer-should-have-shipped-by-now/).