In just a handful of years, focus has progressed from how generative AI can be deployed for assistance in the enterprise to the ways in which more sophisticated tools augment employees' task performance. More recently, the conversation has turned to how it might be possible to replace employees and automate tasks with sophisticated AI agents. In short, we are rapidly moving from assist to augment to automate.
This is where the enterprise is beginning to exploit the significant power of the newest generative AI technologies, with this movement to the right along the automation continuum. However, agentic AI also requires changes in thinking and practice for responsible adoption.
Developing a resilient and secure framework does not need to constrain the AI adoption journey or slow or stifle innovation. Instead, a proper foundation can enable safe and effective AI implementations that meet the organization's needs and make the enterprise more productive and competitive.
And a comprehensive foundational approach to securing agentic AI systems is critical because we are seeing specific emerging threats, such as: poisoning attacks (compromised training data or models that misdirect autonomous agent behaviour); excessive agency (agents acting beyond intended scope to result in unauthorized information access and disclosure); agent collusion (multiple agents coordinating to bypass controls or generate harmful outputs); tool exploitation (agents misusing connected APIs or tools to escalate privileges); and many others.
Cutting-edge generative AI technologies and agents bring novel risks, to be sure. These systems can change subtly over time and are designed to learn continually from prompts and new data. Because generative AI systems learn from these inputs, they continually refine their output—mostly out of anyone's control—spawning issues that have never been seen before. Cybersecurity programs must respond with greater emphasis on observing and evaluating system behavior in real-world environments.
Organizations that can perpetually transform and adapt to keep ahead of rivals are clear about the need to shift right and find greater automation. Pursued responsibly, this is how enterprises will extract maximum value from the advent of new agentic GenAI systems.
A proactive approach to effective governance and architecture, while defining the appropriate data policies and guardrails, provides the enterprise with a secure framework for AI adoption.
The enterprise that starts by defining and adopting a proper framework and embracing a secure-by-design approach will find that its AI journey fosters trustworthiness and resilience. Too often, the experts get called in for a risk assessment and a threat modeling exercise as a new generative AI tool gets close to production, it quickly becomes clear that certain bedrock fundamentals are missing. The further you get down the road, the more difficult it becomes to retrospectively fix fundamental design gaps and put necessary measures in place.
Strong governance must be established, data security and provenance examined, guidelines developed, and security controls mapped for everyone to follow—all early in the journey. Sometimes, a proof-of-concept AI implementation reaches the point where threat modeling is needed prior to developing a full AI adoption framework. Completing the threat exercise can itself be a useful undertaking, helping to demonstrate and inform the requirements of a comprehensive framework that drives a strong and resilient AI and GenAI security posture.
There are multiple security challenges and possible points of failure in agentic AI systems that must be protected against. These include:
Leadership may want to ensure that their AI adoption framework is aligned with existing standards such as the National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) or the International Standards Organization (ISO) 42001 standard. However, they will benefit from going deeper and creating a more contextual framework that adds prescriptive controls and explicitly covers runtime assurance for emerging areas like agentic behaviour. Third parties can help the organization create a more sophisticated and thorough AI security framework.
The secure-by-design concept is not new, of course. However, some treacherous regions in the AI landscape reinforce the need for a deliberate, proactive approach to implementation. Critical areas, such as managing training data for large language models, the integrity of fine-tuning processes, and handling agent privileges and memory, among other examples, can reveal gaps in governance or flaws in architecture. These may result in vulnerabilities and later problems that are difficult to solve.
Because generative AI applications learn and change over time, the need for effective continuous monitoring and dynamic response becomes increasingly important.
Outcomes in generative AI systems can drift over time. This can happen benignly, due to changes in the nature of the data the application works with. Or drift may occur because of the excessive autonomy of a system, or due to poisoned or contaminated data, malicious or low-quality inputs that shift a model’s behavior. The cumulative effect of harmful or biased prompts, injected into the system over time, can also change the model’s behavior.
Without sophisticated monitoring, it can be challenging to catch the inflection point where agents begin to drift autonomously, act with malicious intent, and or produce harmful outcomes. Fortunately, advanced technologies are emerging to track model drift and surface anomalous behavior that may indicate a problem.
A robust approach combines continuous monitoring with corrective adaptation—enabling effective real-time detection and response. One emerging area is so-called dynamic unlearning, where generative AI systems can be made to forget malicious or sensitive data that compromises performance or impacts regulatory compliance. Monitoring and corrective actions together can ensure that harmful behaviors are detected. They can help to eliminate the possibility that sensitive or poisoned data will result in inappropriate information disclosure, misinformation, or repeated biased outputs. Such steps are necessary to ensure compliance with requirements like the right to be forgotten in stringent regulations like the GDPR.
The regular assurance and security testing that is standard practice for most organizations will continue to be essential and still needed across both existing technologies and new AI applications. When a significant change or update is being made or something is being taken to production, a secure framework demands penetration testing, standard input and output filtering, red teaming, and more. These practices remain a critical complement to newer approaches, ensuring enterprises do not lose sight of foundational security disciplines as they adopt AI at scale.
Enterprises need to be nimble, ready to embrace new technology solutions as they appear, to keep AI systems safe. In AI, the foundation is always the data. The security and privacy issues around this data are multifaceted, and the tools to address these issues are evolving. For example, federated learning holds promise for enterprises that want to protect data privacy while training models collaboratively across environments for AI tasks. Differential privacy techniques, which introduce calibrated random variation into training data to defend against inference attacks, are another critical safeguard.
The evolution in AI security is continuous, and enterprises must relentlessly adapt to stay ahead and keep pace with fast-changing best practices.
Creating a brokerage layer allows advanced AI agents to be continuously monitored, ensuring they act as intended and follow established guidelines.
New challenges emerge as enterprises move to the right on the automation continuum, especially from the augmentation of work tasks to more autonomous agentic AI systems. The risk with more advanced agentic systems comes when they become object-oriented or task oriented. At this point, oversight is essential to prevent unsafe, unintended, or non-compliant behaviors.
The best approach here is to ensure that all transactions of agentic AI systems are orchestrated through a broker. By creating a brokerage layer—a message queue or gateway—whenever an AI agent interacts with data, calls an API, or takes other actions, its moves can be monitored and managed. When an agent’s behavior changes, accessing a new data source or a higher-trust system, the activity can immediately be flagged for investigation.
You can’t control what you can’t see. A brokerage layer provides visibility, making it possible to orchestrate the behavior of AI agents and implement adequate controls. A brokerage layer makes continuous monitoring possible to ensure that the agent stays within the periphery of its defined tasks, staying in bounds. There is no better way to manage this large agentic AI ecosystem than to put an orchestrator at its center.
Enterprises must leverage the full range of natively available cloud security offerings, complemented by existing best-of-breed security tools, and mitigate critical residual AI and generative AI risk with specific security tools.
Most enterprises work in one or two large cloud provider environments for their AI systems development. Importantly, this provides the organization access to advanced security controls as part of the package. Microsoft Azure has an AI security posture management toolset, complemented by its AI Content Safety offering and other controls. Google has an entire set of AI security tools under Google AI protection.
These tools cover identity and access management, data privacy, filtering against unwanted outputs, and more. However, they only work if they are fully leveraged and set up properly, which many organizations fail to do. The first step is to ensure that the natively available security controls offered by the cloud provider are fully utilized and effectively deployed.
The next increment of protection comes from leveraging existing best-of-breed security tools that come from third parties and have already been implemented in the environment. These tools provide a range of features such as AI discovery, data protection, run-time protection, and AI security posture management. This is a matter of leveraging the existing investment, and it will get the enterprise to a fair degree of maturity. Still, even after using the native hyperscaler and existing best-of-breed tools, there will be a significant AI-specific residual risk.
The final layer of protection must come from niche and specialized AI-specific security tools. Experts and consultants tailor these to a specific environment or system. Mitigating critical risks related to emerging agentic systems requires precise generative AI-related security tooling, which can contextually protect and deliver more robust threat detection and response. At this level, you are, for example, mapping to MITRE’s ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems) or performing testing against the OWASP (Open World Wide Application Security Project) top ten generative AI threats.
Throughout the process of developing and managing a robust AI and GenAI security framework, from developing AI governance principles to leveraging the right tools to close security gaps, the enterprise needs to be both committed and agile. Organizations must also know how to find the right technology and security partners at each step.
Embracing AI’s extraordinary promise means committing to its responsible implementation—and while that presents an extraordinary challenge in a fast-changing field, it is the only path that assures the likelihood of success and security.