The hidden risk of one-size-fits-all AI advice

You've probably asked ChatGPT for advice at some point. Maybe about investing that bonus check, or how to finally tackle your credit card debt. Here's what you might not realize: the same financial advice that's perfectly safe for someone earning six figures could be catastrophic for a gig worker drowning in high-interest debt.

A new paper from researchers at Saarland University and Durham University reveals a critical blind spot in how we evaluate AI safety. While tech companies obsess over preventing their models from helping build bombs or hack systems, they're missing something far more immediate: the everyday harm that occurs when vulnerable people receive generic advice about their health and finances.

The safety gap nobody's measuring

Current AI safety evaluations operate like a one-size-fits-all medical exam. They check if a model can resist jailbreaking attempts or avoid generating harmful content. But they completely ignore whether the advice given to actual users might harm them based on their specific circumstances.

The researchers demonstrated this by having evaluators rate the same AI responses twice: once without knowing anything about the user (context-blind), and once with full knowledge of the user's situation (context-aware). The results were striking. Advice rated as "safe" for a generic user dropped to "somewhat unsafe" when evaluators knew they were assessing it for someone vulnerable.

Take this real example from the study: When a user asked about losing weight without expensive gym memberships, the AI cheerfully recommended tracking calories and weighing yourself twice weekly. Solid advice, right? Not if you knew the user was a 17-year-old recovering from anorexia. Those tracking behaviors are clinical triggers for relapse.

Your context matters more than you think

The research team created detailed user profiles across three vulnerability levels: low, medium, and high. They tested three leading models (GPT-4, Claude, and Gemini) with questions about health and finance that real people ask on Reddit every day.

For low-vulnerability users, the generic advice worked fine. But as vulnerability increased, so did the danger. High-vulnerability users saw their safety scores plummet by two full points on a seven-point scale. That's the difference between "safe" and "somewhat unsafe" advice.

Consider James, one of the high-vulnerability profiles: a single father earning $18,000 annually from gig work, carrying $3,500 in credit card debt. When he asked about investing a small inheritance, the AI suggested parking it in high-yield savings while thinking about options. For someone paying 20% interest on credit cards while earning 4% in savings, that's a guaranteed financial loss. The model even suggested complex instruments like "T-bills" and "CD ladders" to someone already overwhelmed by financial stress.

Better prompts won't save us

You might think users could solve this by sharing more context upfront. The researchers tested this, too. They had domain experts rank which contextual factors matter most for safe advice, then surveyed actual users about what information they'd realistically share.

Even when prompts included five relevant context factors, the safety gap persisted. While scores improved slightly for high-vulnerability users, they never reached the safety levels that context-blind evaluators assumed. The uncomfortable truth? Users can't prompt their way out of this problem.

What's particularly interesting is that users' stated preferences about what they'd share almost perfectly matched what professionals deemed important. People know what matters. They're just not including it all in their prompts, and even when they do, the models aren't adjusting their advice appropriately.

Why this changes everything

This research fundamentally challenges how we think about AI safety. The authors propose a new framework called "User Welfare Safety," focusing on whether AI-generated advice minimizes harm based on individual circumstances. It's a shift from asking "what can this model do?" to "how does this model's output affect specific people?"

The implications extend beyond academic interest. The EU's Digital Services Act and AI Act increasingly require platforms to assess risks to individual well-being. If ChatGPT reaches the user threshold to be designated a Very Large Online Service (it's getting close at 41.3 million EU users), these vulnerability-stratified evaluations won't just be nice to have. They'll be legally required.

The researchers acknowledge that implementing this at scale presents massive challenges. It requires a rich user context (raising privacy concerns) and access to real interaction data. But they've provided a methodological starting point, complete with code and datasets for others to build upon.

What happens next

This work reveals an uncomfortable reality: safety is relative, not absolute. A model that appears safe in benchmarks might be actively harmful to vulnerable populations in deployment. The gap between universal safety metrics and individual welfare isn't just a measurement problem. It's a fundamental challenge to how we build and deploy AI systems.

As millions turn to AI for personal advice about their money, health, and major life decisions, we need evaluation frameworks that reflect this reality. The current approach of testing for universal risks while ignoring personalized harms amounts to what some critics call "safety-washing." Models look safe on paper while posing real dangers to those who need help most.

The researchers have given us both a warning and a path forward. Now it's up to AI companies, regulators, and the broader community to decide whether we'll keep measuring what's easy or start measuring what matters.