Let me share something that might surprise you: up to 70% of Retrieval-Augmented Generation (RAG) systems fail in production. Yes, you read that right. While RAG looks magical in demos and proof-of-concepts, the reality of production deployment tells a very different story.

I'm Shubham Maurya, Senior Data Scientist at Mastercard with eight years of experience building AI solutions. Throughout my journey developing everything from consumer credit solutions to multi-agent systems, I've seen firsthand how RAG can go from hero to zero when it hits the real world. Today, I want to walk you through the challenges we've faced and, more importantly, how we've solved them.

What exactly is RAG (And why should you care)?

Before diving into the problems, let's get clear on what RAG actually means. RAG stands for Retrieval-Augmented Generation - three simple words that pack a powerful punch:

  1. Retrieve information from your data sources
  2. Augment that information into your prompts
  3. Generate responses using Large Language Models (LLMs)

Think of it this way: when someone asks your system "What's the authorization rate for country XYZ?", RAG finds the relevant information from your databases, adds it to the prompt, and then lets the LLM generate an accurate, grounded response.

But why not just use powerful LLMs?

You might be wondering - with models like GPT-5 and Claude available, why bother with RAG? Here's the thing:

Your data changes constantly. We have clients whose data refreshes weekly. Pre-trained models simply can't keep up with that pace of change.

Domain-specific knowledge matters. Ask ChatGPT about Mastercard's specific transaction decline codes, and you'll get a blank stare. It doesn't have access to our internal information.

Retraining is expensive and impractical. You can't continuously fine-tune LLMs every time your data changes. It's costly, time-consuming, and requires significant expertise.

Explainability is crucial. With RAG, you can see exactly what information was retrieved and why certain answers were generated. Try explaining a pure LLM's output - good luck with that!

Explainability and transparency in autonomous agents
AI experts share strategies for transparency, governance, and trust in agentic systems, plus techniques for building explainable, auditable models.
AI Accelerator InstituteSaradha Nagarajan

The four horsemen of RAG failure

Let me walk you through the four main challenges that cause RAG systems to fail in production:

1. Knowledge drift: When yesterday's truth becomes today's lie

Here's a real example: You build a RAG system when interest rates are 4%. Six months later, they've jumped to 5.5%. But your system? It's still confidently telling users the rate is 4%.

Or consider what happened to us at Mastercard. We had a massive transaction table that we decided to split into domestic and international transactions. Our text-to-SQL solution kept trying to query the old table that no longer existed. Result? Errors everywhere.

2. Retrieval decay: Death by data growth

In your POC with a small dataset, retrieval works beautifully. Fast forward six months, and you've got millions of documents. Suddenly, your system can't find the needle in the haystack anymore.

We experienced this firsthand when trying to find top merchants and merchant codes. The system would retrieve the same redundant information multiple times, missing crucial details because of our context size limits.

3. Irrelevant chunks: The information overload problem

Imagine asking for a simple definition and getting a 10-page dissertation in response. That's what happens when your retrieval brings back too much irrelevant information. LLMs, just like humans, get confused and start hallucinating when overwhelmed with data.

4. The evaluation gap: Flying blind

This might be the most painful challenge. Have you ever given feedback using those thumbs up/down buttons in ChatGPT? Exactly - nobody does. So how do you know if your production RAG system is deteriorating? By the time users lose trust and stop using it, it's already too late.

How we fixed these problems (And how you can too)

Smarter retrieval strategies

Hybrid search: We don't just rely on semantic search or lexical search - we use both. When someone asks about "ISO 8583 field 55 definition," lexical search finds the exact match. For broader questions, semantic search understands context. The magic happens when you combine them.

Graph-based RAG: For our text-to-SQL solutions with multiple interconnected tables, traditional RAG would miss crucial joining conditions. Graph-based retrieval understands relationships between tables, dramatically reducing errors and hallucinations.

Making RAG aware of changes

We developed a schema evolution tracking system. When our transaction table split into domestic and international tables, our RAG system automatically detected this change. Now, when users query transactions, the system knows about the new structure and generates correct SQL queries.

This approach works for any evolving information - from changing interest rates to updated privacy definitions post-COVID.

Performance optimization

With 10 million records in our vector database, retrieval was painfully slow. Our solution? Intelligent segmentation. When someone asks about column definitions, we only search the schema segment. Analytics questions? We search the analytics segment. Response times dropped dramatically.

Adaptive context sizing

Not all questions need the same amount of context. Asking for "top 5 science fiction books"? We retrieve maybe 10-15 documents. Asking for "all available science fiction books"? We adapt and retrieve much more.

We use LLMs to detect user intent and adjust retrieval accordingly. It's not one-size-fits-all anymore.

Smart summarization

Before feeding retrieved information to the LLM, we summarize it. Think about it - would you prefer a three-page document or a concise paragraph answering your question? LLMs are the same. This not only improves accuracy but also reduces costs by using less of the LLM's context window.

Continuous feedback loop

Instead of relying on user feedback (which rarely comes), we:

  • Log all user queries
  • Use libraries like RAGAS to evaluate retrieval quality
  • Check for groundedness, relevancy, and hallucinations
  • Generate synthetic test data based on real queries
  • Re-evaluate and fine-tune monthly or weekly
Copilot vs Claude for Excel: Which AI assistant wins for formula building?
We’ll talk about Claude and Copilot & compare and contrast how these two AI assistants help you with building spreadsheets and writing formulas.
AI Accelerator InstituteSalvatore Tirabassi

Real-world applications that actually work

These aren't theoretical solutions. We've successfully implemented them in:

  • AI governance: Automating documentation for EU compliance using policy-based RAG
  • Diagnostic analytics: Multi-agent systems helping customers improve authorization rates
  • Text-to-SQL: Natural language interfaces that correctly query complex databases
  • Automated testing: Tools that write unit tests and run pipeline checks

The future of RAG

Looking ahead, I see three exciting developments:

  1. Self-retrieving LLMs: Where retrieval becomes just another tool the LLM can use autonomously
  2. Graph + RAG integration: Deeper integration for handling complex, interconnected data
  3. Multi-agent orchestration: Systems that know when they need more information and automatically retrieve it

The bottom line

RAG in production is hard, but it's not impossible. The key is understanding that what works in a demo rarely scales to production without significant adaptation. By implementing smarter retrieval strategies, making your system aware of changes, optimizing performance, and creating continuous feedback loops, you can build RAG systems that actually deliver on their promise.

Remember: every failed RAG system is an opportunity to learn and improve. The challenges are real, but so are the solutions. Start with understanding your specific use case, implement these strategies incrementally, and always keep measuring and adapting.

Because at the end of the day, a RAG system that works 70% of the time in production is infinitely more valuable than one that works 100% of the time in a demo.