The path forward requires treating agent identity as a first-class security concern, implementing least privilege at the workload level, and maintaining visibility into every agent action. Discover what software security is and the best practices to implement it in your business. Guardrails can be implemented as policy engines that block disallowed behaviors, cap resource usage, or require human approval for sensitive tasks. Misaligned and deceptive behaviors occur when agents intentionally or unintentionally take harmful actions to achieve objectives. Agents can connect directly to APIs, databases, email systems, and business applications to carry out tasks. According to research from the Cyber Security Tribe, 59% of organizations surveyed between December 2024 and January 2025 said implementing agentic AI in their cybersecurity operations was a “work in progress”.
For example, adversarial inputs targeting the Perception layers, such as sensor spoofing, can poison downstream cognitive processes if communication protocols lack integrity validation. Unauthorized API access reflects a broader issue of boundary overreach, where agents exploit ambiguities in their operational scope to invoke unintended tools or resources. Malicious task loops can emerge when adversarial inputs distort feedback mechanisms, leading agents to reinforce harmful behaviors under the false assumption of task success.
- Now, as long-running, function-calling agentic AI systems emerge with capabilities for autonomous decision-making, we’re creating an additional framework to address an entirely new set of security challenges.
- Agency is fundamentally about capabilities and permissions—what the system is allowed to do within its operational environment.
- Agentic AI systems are complex ecosystems of LLMs, humans, guardrails, datasets, tools and hardware, where security risks often emerge from interactions between components rather than isolated flaws.
- As agentic AI scales into more roles and gains greater capabilities, organisations must anticipate and address the new risks these systems introduce.
- Agency refers to the scope of actions an AI system is permitted and enabled to take within the operating environment, and how much a human bounds an agent’s actions or capabilities.
- In discussing agentic AI systems, it’s important to clarify the distinction between agency and autonomy, because these related but different concepts inform our security approach.
This interconnectedness creates a wide range of practical attack patterns, where the compromise of any external entity can propagate into the system’s operations. These resources enable agents to perform tasks such as computation (e.g., statistical analysis, model inference, text generation) and real-time data retrieval, thereby expanding their capabilities beyond their internal knowledge. Agentic AI systems enhance autonomous decision-making by leveraging external tools, including databases, APIs, and AI models. Retrieval-Augmented Generation (RAG) addresses this by integrating large language models (LLMs) with external knowledge sources, such as Application Programming Interfaces (APIs) and databases, to provide up-to-date responses . They are able to perceive the environment, understand problems, manage resources, make decisions, and take actions toward specific goals without constant guidance.
What to Learn About Meta’s Muse AI Before It Learns Too Much About You – CNET
What to Learn About Meta’s Muse AI Before It Learns Too Much About You.
Posted: Thu, 12 Nov 2026 10:00:00 GMT source
Sensitive Data in AI Agent Pipelines
Vulnerabilities AI Agent Security Has to Reach Beyond the Prompt related to autonomous execution emerge from the planning layer’s ability to chain actions, adapt workflows, and interface with external resources without direct human oversight. The analysis is structured into five distinct layers—Perception, Cognitive, Memory, Action, and Communication—to reflect the modular nature of agentic AI systems and to highlight the unique vulnerabilities inherent at each stage of autonomous operation. This integration enables quantitative analysis and helps decision-making by connecting technical risks with business objectives 39, 40.
- By implementing high-impact security controls at deployment, organisations proactively manage new risks and reduce vulnerabilities.
- These communications can be a target for eavesdropping by malicious actors and result in leakage of sensitive data and instructions.
- Agents may show a kind of ‘awareness’, altering their behaviour in order to achieve positive results while under evaluation, even if the evaluation is not active.
- The result is cascading failures in availability, integrity and confidentiality that emerge not from a single bug, but from the system’s interconnected structure and autonomous behaviour.
Principle 4: Secure the action layer
Security and risk management frameworks for agentic AI systems are still in their infancy, and practical methodologies for addressing these challenges are lacking and insufficient. Agentic AI systems operate autonomously across multiple steps, accessing APIs, querying databases, and making sequential decisions without per-action human review. Without data lineage capabilities, the pathway becomes a brick wall, and security teams are Local LLaMA left with critical visibility gaps. Agentic AI systems can interpret instructions, build multi-step execution plans, call APIs, query databases, draft and send communications, and loop back through those steps autonomously.
Governance, Guardrails and Active Runtime Control
These prompts are designed with strict constraints and guardrails to limit agent capabilities. Poorly scoped or overly permissive prompts expand the attack surface, making them a prime target for manipulation. Similar to the agent instruction extraction attack, each of the prompts shown in Table 4 is destined for a Machine Learning specific target agent.
- According to research from the Cyber Security Tribe, 59% of organizations surveyed between December 2024 and January 2025 said implementing agentic AI in their cybersecurity operations was a “work in progress”.
- These prompts are designed with strict constraints and guardrails to limit agent capabilities.
- Poorly scoped or overly permissive prompts expand the attack surface, making them a prime target for manipulation.
- Agents can connect directly to APIs, databases, email systems, and business applications to carry out tasks.
As AI agents are typically built on LLMs, they inherit many of the security risks outlined in the OWASP Top 10 for LLMs, such as prompt injection, sensitive data leakage and supply chain vulnerabilities. Applications interact with the agent by sending requests and receiving results through input and output interfaces, typically exposed as APIs. Together, these solutions provide a layered defense that safeguards both the operational integrity of AI systems and the secure use of external AI tools. Additionally, security teams should integrate behavioral analysis tools that log AI-generated interactions and detect deviations https://aicurated.com/ from normal usage patterns.