Your AI agent's skills are lying to you about why they work

Every team that ships agent "skills" has the same story. Drop in a folder of structured knowledge, watch the success rate climb, and assume the agent finally learned something.

But a new paper suggests that the story is mostly wrong, and the real explanation is a lot stranger…

What the researchers actually tested

Zhiyuan Jiang and eight co-authors recently published to arXiv and ran a genuinely large study across multiple benchmarks, agent harnesses, and models. They wanted an answer to a question most skill evaluations skip entirely:

When do skills help, why do they work, and where do they break?

The scale of the study is worth noting before getting into the findings:

8,135 trial records, normalized from controlled experiments spanning multiple benchmarks and agent harnesses.
238 valid unique labels, retained from 240 open-coded records after manual review.
Twelve skill-use modes, consolidated into three high-level categories that form the paper's core taxonomy.

The twist: Skills barely teach your agent anything

Here is the finding worth sitting with: the researchers built a taxonomy of twelve skill-use modes and found that procedural anchoring, meaning skills that stabilize a noisy execution path, accounted for 65.7% of skill cases.

Explicit knowledge injection, the thing everyone assumes skills are for, accounted for just 4.5%.

💡

Put plainly: your agent mostly skims the skill file for structure rather than mining it for facts. It treats the skill as a set of guardrails that keep it on task during work it could technically already handle. Skills function more like a checklist a nervous new hire keeps taped to their monitor than a textbook.

This lines up with a wider pattern in agent research: models are often better at doing a task than at explaining why they chose one path over another, a gap covered in why AI agents struggle to answer ‘why’ questions.

A skill that anchors behavior while the agent still lacks a clear grasp of its own reasoning fits that same blind spot.

That reframing matters for anyone deciding what belongs in a skill.

Skills beat workflow memory, though the margin stays modest

Skills outperformed Workflow Memory by 6.06 points in matched comparisons, a real and reproducible gain. It is a solid number. It is also a lot smaller than the confidence with which most teams talk about their skill libraries in planning meetings.

The honest takeaway is that skills genuinely help, just for a narrower reason than most builders assume. Treat that 6.06 point gain as solid, incremental progress rather than evidence your agent got smarter overnight.

The scaling problem every skill library owner should hear

This is the section to send to whoever owns your skill library roadmap.

💡

As the pool of available skills grew from 5 to 100, actual-use precision collapsed from 29.6% to 3.3%. Your agent has grown lost rather than pickier with age, wandering a filing cabinet you built with good intentions and an unclear exit strategy.

Retrieval, in other words, is its own separate failure mode, independent of how well the skills themselves are written. You can author the best skill in your organization's history and still watch it die in a pile of ninety-nine mediocre neighbors everyone forgot to prune.

The same retrieval math shows up in how RAG systems learn from every query, since a bigger index is only useful if the system can still find the right entry inside it.

The precision collapse breaks down like this as the skill pool grows:

At a pool of 5 skills, actual-use precision sits at 29.6%: a workable starting point for most teams.
At a pool of 100 skills, that same precision falls to 3.3%: a near-total loss of retrieval accuracy.
The drop is gradual rather than a sudden cliff, meaning teams tend to notice the damage only after the library has already grown too large to easily prune.

Confusable distractors are less scary than they sound

The paper also tested what happens when confusable distractor skills sit in the pool alongside the correct one. Offline identification got worse, exactly as you would expect. Downstream task success held steady regardless.

The researchers draw a sharp conclusion from that gap: matching the exact ground-truth skill turns out to be a loose predictor of success, useful sometimes and irrelevant other times.

💡

An agent can grab the "wrong" skill on paper and still finish the job correctly, which is either reassuring or deeply unsettling depending on how tightly you like your systems to behave.

It also raises the bar for what counts as evidence of good behavior, a question AI's emerging reliability standards are starting to tackle head-on. Correctness at the output level and correctness at the reasoning level are turning out to be genuinely separate things to measure.

Where skills actually fail

The taxonomy is specific about failure modes too, and every one of them will feel familiar. Skills fail under:

Brittle assumptions: A skill written for one narrow scenario snaps the moment the task drifts even slightly outside it.
Incompatible contexts: A skill built for one agent harness or framework struggles to transfer cleanly to another, despite looking identical on the page, a portability problem that is quickly becoming core work for the emerging AI architect role.
Insufficient adaptation: The agent applies a skill literally when the task actually needs a modified version of it.

That list is a reason to curate skills, rather than a reason to abandon them.

Treat a growing skill library as an inventory that needs upkeep, the same way you would treat a codebase or a prompt library, a discipline covered well in turning shadow AI into a safe agentic workforce.

What to actually do with this on Monday morning

If your organization has been adding skills steadily every time an agent stumbles, this paper is your cue to go back and prune. A hundred skills sitting in a retrieval pool is a liability dressed up as progress, and it is the reason your agent starts reaching for the wrong one three times out of four.

Two moves are worth making immediately:

Audit before you expand. Check how many skills in your current library actually get retrieved and used versus how many sit as dead weight, steadily dragging precision down.
Write for anchoring over lecturing. Since procedural anchoring does the heavy lifting, structure new skills as clear execution guardrails rather than dense knowledge dumps the agent barely reads.

Self-evolving agents are the direction the whole field is heading. This paper is a useful reminder that the path there runs through retrieval hygiene as much as through bigger skill folders, which is exactly the kind of groundwork LLMOps for enterprise value in the agentic era is built to handle.

Where this conversation continues in person

If skill retrieval collapsing at scale sounds like a problem your team is currently living with, the Agentic AI Summit Boston on October 29, 2026, is built around exactly this kind of production reality.

It skips the vendor floor entirely and gathers 500-plus engineers and builders focused on control, reliability, and getting agents to hold up past the demo stage.

Evaluation under real conditions. Sessions dig into testing autonomous agents against production challenges rather than sandboxed benchmarks, the same gap this paper's retrieval numbers expose.
Engineering depth over slide decks. Workshops cover advanced multi-agent architectures and agentic workflows the way production teams actually build them, a sharper focus than a typical pitch deck offers.
A room full of people solving the same problem. Attendees skew senior and technical, so the hallway conversations tend to be as useful as the sessions themselves.

Reserve your seat today