CausalDS: the AI benchmark that tests real causal reasoning

A new benchmark out of the University of Michigan just handed six frontier AI agents a causal reasoning exam. The scores expose a gap that raw capability metrics tend to paper over.

The benchmark is called CausalDS, published on arXiv on July 9, 2026, by Andrej Leban and Yuekai Sun. It tests something more specific than intelligence.

Can these agentic systems reason about cause and effect, or do they just sound like it?

The timing is good, if a little inconvenient for the industry...

Every vendor markets its agents as data-science collaborators these days. CausalDS asks the harder question underneath that pitch:

When the answer depends on separating correlation from intervention, does the agent actually get there?

Why data science agents keep failing causal reasoning

Leban and Sun point to a split in existing benchmarks. Symbolic causal reasoning tests, like CLadder, check a model's logic over an abstract graph.

Data-science agent benchmarks, like DSBench and MLE-bench, test a model's ability to write code and produce a defensible result.

Very few tests cover both at once.

The ones built from curated real-world datasets also carry a contamination risk. A model can score well by recalling a familiar example instead of reasoning through the actual structure.

Researchers have a lovely name for that shortcut:

The "causal parrot" problem.

CausalDS closes the gap by generating everything from scratch.

Each scene starts with a randomly sampled structural causal model, then the researchers generate observational data from it.

An LLM translates the abstract graph into a domain story, complete with named variables and units.

A farming subsidy, an irrigation decision, a soil erosion outcome, all invented for the occasion and dressed up convincingly. The graph is real and stays hidden. An audit loop keeps the true answer locked away, so early guessing earns a model exactly zero credit.

Inside the exam

From each scene, the benchmark derives tasks spanning all three rungs of Pearl's causal hierarchy: association, intervention, and counterfactual reasoning.

An agent might predict an outcome from raw data, decide if a treatment effect is even estimable, or flag a proposed adjustment set as biased.

💡

Some questions are built so any confident answer would be unwarranted given what was released. Recognizing that moment and abstaining counts as its own scoring skill, separate from getting the underlying math right. Knowing when to shrug turns out to be genuinely hard.

The researchers also added a data-quality dial. Some scenes hand the agent clean measurements; others swap key variables for noisy proxies and make it work for the answer.

That separates a model's grasp of causal structure from its ability to survive the messy data every working analyst deals with daily.

The leaderboard

On a 100-scene realistic exam drawn from a pool of 953 total scenes, Claude Opus 4.8 topped the table. It landed a CausalDS score of 0.278 and an 82.4percent pass rate.

Gemini 3.1 Pro placed second at 0.370.

Qwen 3.6 35B and Kimi K2.6 followed, then GPT-5.5, with Gemma 4 26B trailing the field at 0.644. The full board, for anyone who likes their evidence in table form:

Claude Opus 4.8: score 0.278, 82.4% pass rate, 17.7k tokens per task
Gemini 3.1 Pro: score 0.370, 76.5% pass rate, 145.6k tokens per task
Qwen 3.6 35B: score 0.447, 63.2% pass rate, 140.7k tokens per task
Kimi K2.6: score 0.475, 65.7% pass rate, 266.4k tokens per task
GPT-5.5: score 0.561, 82.4% pass rate, 12.9k tokens per task
Gemma 4 26B: score 0.644, 55.9% pass rate, 32.4k tokens per task

The token column is where things get genuinely funny.

Claude Opus 4.8 reached its leading score on 17.7k tokens per task, while Kimi K2.6 burned through 266.4k tokens for a worse result, a fifteen-fold gap for less to show for it.

GPT-5.5 matched the leader's 82.4% pass rate using the fewest tokens on the board, yet finished fifth overall. Efficient reasoning and accurate reasoning are apparently two separate skills, a distinction most benchmark marketing tends to gloss over.

Gemma 4 26B deserves its own paragraph. It submitted a valid answer on every single task, a perfect completion rate, and still finished last. In casual work, that combination usually means one thing: the model answered when the wiser move was to pause, like the colleague who fills every silence in a meeting with confident nonsense.

Five axes, one score

CausalDS scores every agent across five separate capabilities, and the paper's central finding is that these move independently. A model strong on one axis can land weak on another, which is exactly why a single leaderboard number invites the wrong conclusion.

Symbolic causal reasoning: separating associational claims from interventional and counterfactual ones.
Data-science execution: writing and running code that actually produces an estimate from tabular data.
Uncertainty quantification: attaching a calibrated interval to an estimate instead of a falsely precise point value.
Epistemic abstention: recognizing when the data genuinely fall short of an answer, then abstaining rather than hallucinating a confident one.
Tool use and coding efficiency: reaching a correct answer inside a sandboxed environment while keeping steps and token spend in check.

What this means for teams evaluating agentic AI

Any team piloting agents for causal inference, marketing mix modeling, or root-cause analysis should treat CausalDS as a preview of the failure modes waiting in production.

💡

The paper's abstention framing is the most transferable idea here, and it travels well beyond causal work.

An agent that always answers signals confidence, but confidence and reliability are different currencies. A vendor demo that only rewards raw accuracy tends to hide that distinction from the people making the purchasing decisions.

Leban and Sun frame this nicely in the paper. A related benchmark, CausaLab, tests an experimental discoverer.

CausalDS tests a genuine causal data scientist working from a pile of files and a deadline. That is the role most enterprise teams actually need filled.

A few questions worth asking before trusting an agent with a real causal claim:

Ask vendors for pass rate and abstention rate separately, rather than a single blended accuracy figure a confidently wrong model can inflate.
Test agents against deliberately non-identifiable questions, the causal equivalent of a trick question, before trusting output on ambiguous real-world data.
Track token cost per correct answer. CausalDS shows some models pay for accuracy with budgets many times larger than a rival hitting a similar pass rate.

The causality test that humbled six AI agents