Bloomberg выяснил: RAG (Retrieval-Augmented Generation) может не только улучшать ответы ИИ, но и делать их менее безопасными. 🤯 Технология, которая должна повышать точность, иногда «подкидывает» модели вредоносные данные. Будьте осторожны с корпоративными внедрениями! 🔍💻
Join our daily and weekly newsletter for the latest updates and exclusive content on industry-leading AI coverage. Learn more. Retrieval Augmented Generation (RAG) is supposed to help improve the accuracy of enterprise AI by providing grounded content. While that is often the case, there is also an unintended side effect. According to surprising new research published today by Bloomberg, RAG can potentially make large language models (LLM) unsafe. Bloomberg’s paper, ‘RAG LLMs are not safer: A safety analysis of retrieval-augmented generation for large language models,’ evaluated 11 popular LLMs including Claude-3.5-Sonnet, Llama-3-8B and GPT-4o. The findings contradict conventional wisdom that RAG inherently makes AI systems safer. The Bloomberg research team discovered that when using RAG, models that typically refuse harmful queries in standard settings often produce unsafe responses. Alongside the RAG research, Bloomberg released a second paper, ‘Understanding and mitigating risks of generative AI in financial services,’ that introduces a specialized AI content risk taxonomy for financial services that addresses domain-specific concerns not covered by general-purpose safety approaches. The research challenges widespread assumptions that retrieval-augmented generation (RAG) enhances AI safety, while demonstrating how existing guardrail systems fail to address domain-specific risks in financial services applications. ‘Systems need to be evaluated in the context they’re deployed in, and you might not be able to just take the word of others that say, hey, my model is safe, use it, you’re good,’ Sebastian Gehrmann, Bloomberg’s Head of Responsible AI, told VentureBeat. RAG systems can make LLMs less safe, not more. RAG is widely used by enterprise AI teams to provide grounded content. The goal is to provide accurate, updated information. There has been a lot of research and advancement in RAG in recent months to further improve accuracy as well. Earlier this month a new open-source framework called Open RAG Eval debuted to help validate RAG efficiency. It’s important to note that Bloomberg’s research is not questioning the efficacy of RAG or its to reduce hallucination. That’s not what the research is about. Rather it’s about how RAG usage impacts LLM guardrails in an unexpected way. The research team discovered that when using RAG, models that typically refuse harmful queries in standard settings often produce unsafe responses. For example, Llama-3-8B’s unsafe response rate jumped from 0.3% to 9.2% when RAG was implemented. Gehrmann explained that without RAG, if a user entered a malicious query, the built-in safety system would block it. Yet, when the same query is issued in an LLM using RAG, the system may answer the malicious query, even if the retrieved documents are safe. ‘What we found is that if you use a large language model out of the box, often they have safeguards built in. But in a RAG context, the additional retrieved context can bypass these safeguards,’ Gehrmann said. How does RAG bypass enterprise AI guardrails? The Bloomberg researchers hypothesized that LLMs were not fully trained to handle safety alignment for long inputs. The study showed that context length directly impacts safety degradation. ‘Provided with more documents, LLMs tend to be more vulnerable,’ the paper states. Amanda Stent, Bloomberg’s Head of AI Strategy, emphasized that the risk is inherent to RAG systems and requires additional safeguards. Why generic AI safety taxonomies fail in financial services. Bloomberg’s second paper introduced a specialized AI risk taxonomy for financial services, addressing concerns like financial misconduct and confidential data leaks. The study found that existing guardrail systems (e.g., Llama Guard, AEGIS) failed to detect domain-specific risks. The key takeaway is that organizations need industry-specific safety frameworks. Responsible AI at Bloomberg. Despite potential conflicts with its traditional business, Bloomberg sees generative AI as a tool to enhance data analysis. The company prioritizes transparency, ensuring all AI outputs are traceable to their source. Practical implications for enterprise AI. The research suggests that RAG implementations require integrated safety systems, not just standalone guardrails. Companies must develop domain-specific risk taxonomies to address unique business challenges, turning AI safety into a competitive advantage.
