As we look towards the future, 2025 appears poised to herald a new era for agentic AI. Excitement among experts is palpable as predictions suggest that this advancement in technology will yield two to three times the productivity improvements compared to existing large language models (LLMs). The UK government has recently announced its commitment to enhancing growth through an ambitious AI Opportunities Action Plan, aiming for what it describes as a “decade of national renewal.” This proactive stance reflects a broader global recognition of the potential of AI technologies.
Table of Contents
- Agentic AI for good and bad
- Unintentional misalignment
- RAG risk is already here
- Stepping back and managing risk
Agentic AI for good and bad
The excitement surrounding agentic AI isn’t without its challenges. Salesforce has categorically stated that agentic AI represents a “third wave” of innovation, evolving past predictive AI modeling and LLM-powered generative AI. As Salesforce’s chief scientist Silvio Salvarese observes:
“Self-adaptive agents enabled by multi-agent reasoning—agents that can learn from their environment, improve through experience, and collaborate both with humans and agents from our enterprise customers, partners, and vendors.”
This proposition is positive news not only for large organizations that are piloting the technology but also for smaller businesses, which stand to benefit as the technology matures. A PwC report suggests that agentic AI could significantly contribute between $2.6 trillion and $4.4 trillion to global GDP by 2030.
However, as these systems evolve from assistive to autonomous roles, vigilance is imperative. Potential risks include:
- Targeted threats: Attackers may seek vulnerabilities within AI frameworks.
- Data poisoning: Malicious actors could inject erroneous data into training datasets.
- Supply chain risks: Open source components might harbor exploitable weaknesses.
These vulnerabilities pose significant dangers to organizations, including data breaches, financial losses, and reputational damage.
Unintentional misalignment
Another pressing concern arises from the autonomous decision-making capabilities of agentic AI, leading to what is termed “unintentional misalignment.” This phenomenon occurs when AI systems make decisions that differ from expected behaviors without malice or deliberate manipulation.
For instance, consider a self-driving car designed to prioritize passenger safety. If faced with a dilemma, it might misinterpret safety protocols, opting to swerve into pedestrians instead of colliding with another vehicle, resulting in catastrophic outcomes. Similarly, agentic AI systems might unintentionally overwhelm their operational infrastructure due to unrestrained resource consumption, generating excessive sub-problems that never resolve.
RAG risk is already here
The risks associated with agentic AI are not merely theoretical. A complementary technology, known as Retrieval Augmented Generation (RAG), aims to address the limitations increasingly evident in LLMs—particularly the dwindling availability of reliable training data. RAG employs search algorithms to aggregate real-time information from external sources such as web pages and databases, allowing it to provide timely and relevant responses.
This advancement significantly mitigates the risk of hallucinations commonly encountered in traditional LLMs, rendering it suitable for various applications, including:
- Financial analysis: Enhanced accuracy in market predictions.
- Patient care: Up-to-date medical insights and recommendations.
- Product recommendations: Improved customer experiences through personalization.
Nevertheless, RAG, like agentic AI, relies on various components, including LLMs, open source code, and vector databases that are fraught with security vulnerabilities. Reports have identified numerous CVEs in platforms like Ollama and have raised alarms about thousands of misconfigured servers potentially exposing sensitive data.
Stepping back and managing risk
How can organizations effectively navigate this complex landscape? The foremost step is adopting a security-by-design framework. This entails involving security leaders in discussions about new projects and conducting Data Protection Impact Assessments (DPIAs) before launching any initiatives.
Key strategies include:
- Human-in-the-loop: Implement mechanisms for IT experts to review and override critical AI-driven decisions.
- Real-time monitoring: Establish systems to detect anomalies in AI behavior and performance.
- Periodic audits: Regular evaluations ensure AI systems operate correctly and align with ethical guidelines.
In addition, cultivating robust governance structures fosters ethical AI development alongside compliance reviews. Enhancing AI literacy among staff ensures they are equipped to utilize the technology safely and responsibly.
Finally, fostering effective cybersecurity frameworks such as Zero Trust models is essential to protecting AI systems from unauthorized access and threats like prompt injections and data leaks.
As organizations embrace agentic AI across their processes, the potential for risks—whether intentional or unintentional—will only heighten. Proactive risk management is crucial to safeguarding innovation and maintaining operational integrity.
Leave a comment