
AI security: prompt injection, data leakage and model threats, and how to limit the damage
AI security protects the AI systems a company builds or deploys from attacks such as prompt injection, data leakage, poisoning and model theft. This guide maps ten OWASP risks onto five attack layers. It shows which controls limit the damage when an attack succeeds.
What is AI security, and how is it different from AI for cybersecurity?
AI security protects models, data and AI-powered applications against attacks and misuse. It differs from AI for cybersecurity, which uses artificial intelligence to detect threats, analyze suspicious activity or support security teams.
There are three related concepts:
| Concept | Meaning |
| Security of AI | Protecting AI models, applications, data and connected tools |
| AI for cybersecurity | Using AI to improve security operations |
| AI-enabled attacks | Using AI to assist phishing, fraud or other malicious activity |
This guide focuses on security of AI, particularly enterprise applications built around large language models.
These systems introduce risks that conventional application security does not fully address. An attacker may manipulate retrieved documents, influence model behavior or exploit the permissions granted to an AI assistant.
What does the attack surface of an AI system look like?
For this guide, we group the AI application attack surface into five practical layers: prompts, retrieved data, models and dependencies, outputs and tools, and infrastructure. This is a threat-modeling structure used to organize the risks and controls discussed here, not a standardized five-layer AI security taxonomy. Each layer creates opportunities for attackers to influence behavior, access information or trigger unauthorized actions.
Five layers of AI security
| Layer | Typical threat | Main defensive approach |
| Input and prompts | Prompt injection and jailbreaking | Input checks and restricted capabilities |
| Retrieved data | Malicious documents and unauthorized retrieval | Source validation and permission-aware retrieval |
| Model and supply chain | Poisoning and compromised dependencies | Provenance checks and artifact verification |
| Output and tools | Unsafe generated commands and excessive agency | Output validation and least privilege |
| Infrastructure | Credential theft and resource abuse | Identity controls, rate limits and monitoring |
The main difference from conventional software is that LLMs process instructions and external information through a shared language context.
A document retrieved for summarization can contain malicious instructions. If the model follows them, the application may behave differently without any change to its code.
This is why securing the model alone is insufficient. The surrounding application must independently enforce its security boundaries.
What are the main AI security risks according to OWASP?
The OWASP Top 10 for LLM Applications 2026, published in August 2026, identifies ten major risks affecting applications built around large language models. This edition updates the ordering and scope of the previous list, including replacing System Prompt Leakage with the broader category Hidden Context Exposure. The risks provide a starting point for threat modeling and prioritizing security controls.
| OWASP risk | What can go wrong | Key control |
| LLM01:2026 Prompt Injection | Malicious instructions redirect the model | Restrict permissions and tool access |
| LLM02:2026 Sensitive Information Disclosure | Confidential data appears in responses | Data minimization and authorization |
| LLM03:2026 Excessive Agency | AI performs unauthorized actions | Least privilege and human approval |
| LLM04:2026 Supply Chain | Compromised models or dependencies enter the system | Verify origins and versions |
| LLM05:2026 Data and Model Poisoning | Manipulated data changes model behavior | Validate datasets and track lineage |
| LLM06:2026 Unbounded Consumption | Excessive usage exhausts resources or budgets | Quotas and rate limits |
| LLM07:2026 Misinformation | Incorrect answers influence decisions | Verify consequential outputs |
| LLM08:2026 Hidden Context Exposure | Non-user-facing instructions, tool schemas or operational context are exposed | Keep secrets out of model context and enforce authorization independently |
| LLM09:2026 Vector and Embedding Weaknesses | Retrieval exposes unsuitable or restricted content | Enforce retrieval permissions |
| LLM10:2026 Improper Output Handling | Generated output causes downstream vulnerabilities | Validate and safely process output |
Controls are an editorial summary based on OWASP’s 2026 risk catalogue.
Hidden Context Exposure is broader than disclosure of a system prompt. It includes hidden developer instructions, retrieved policy context, tool and function schemas, and other operational information supplied to the model but not intended for users. Applications should assume that such context may be discoverable. Credentials must not be stored in it, and authorization, privilege separation and other security boundaries must not depend on keeping it secret.
GitHub
OWASP describes common application risks, while MITRE ATLAS documents adversarial techniques targeting AI systems. NIST AI 100-2 E2025 provides a taxonomy of adversarial machine learning threats.
Together, these resources help teams identify vulnerabilities, understand attack methods and develop realistic security tests.
How does prompt injection work, and can it be prevented?
Prompt injection occurs when an attacker introduces instructions that redirect an AI application’s intended behavior.
It can take several forms:
- Direct prompt injection: Malicious instructions are submitted directly to the assistant.
- Indirect prompt injection: Instructions are hidden in documents, webpages, emails or tool responses that the assistant processes.
- Multimodal injection: Adversarial instructions are embedded in images or other supported media.
For example, an assistant may retrieve a webpage while researching a legitimate question. The page contains instructions telling the model to ignore its task and reveal internal information.
The attacker wants the model to treat external content as authoritative instructions.
Prompt injection vs jailbreaking
Prompt injection and jailbreaking are related but distinct attack concepts.
Prompt injection introduces attacker-controlled instructions that attempt to override the application’s or developer’s intended behavior. It can manipulate an application without necessarily asking the model to generate prohibited content.
Jailbreaking specifically attempts to bypass a model’s trained safety restrictions, causing it to produce outputs it would normally refuse. The two techniques can overlap or be combined, but jailbreaking is not necessarily a subtype of prompt injection.
OWASP notes that foolproof prompt injection prevention remains uncertain. Filters and prompt instructions reduce risk, but they cannot guarantee that every malicious input will be rejected.
The stronger approach combines detection with controls the model cannot override, including authorization checks, restricted tools and approval for sensitive actions.
How does AI data leakage happen?
AI data leakage occurs when sensitive information becomes accessible through model outputs, retrieved context, training data or connected tools.
Four common routes are:
| Leak route | Example | Prevention |
| Training data | Sensitive examples appear in generated responses | Data minimization and privacy controls |
| Retrieved documents | Assistant accesses files the user cannot open | Permission-aware retrieval |
| Hidden application context | Internal instructions, credentials or sensitive operational details become exposed | Keep secrets outside model context and enforce security boundaries independently |
| Tool outputs | Confidential information reaches external services | Restrict outbound actions |
Why RAG permissions matter
Retrieval-augmented generation (RAG) allows an LLM to use external knowledge. However, a retrieval system may access more documents than the person asking the question.
If the application retrieves confidential information before checking permissions, the model may expose it in its answer.
The solution is to enforce authorization before information enters the model’s context.
Document-level permissions, access isolation and source validation are essential to secure RAG applications.
This article addresses leakage from systems an organization builds or deploys. Employees sharing information through unauthorized AI tools represent a related but separate shadow AI risk.
How do data poisoning and supply chain attacks affect AI?
Data poisoning manipulates information used to train, fine-tune or support an AI system. The objective may be to introduce false associations, alter predictions or create hidden behaviors.
For example, an attacker who can modify an internal knowledge-base document may insert false business instructions. When the assistant retrieves that document, it may repeat the manipulated information.
Supply chain attacks target external dependencies, including model checkpoints, datasets, libraries and packages.
Organizations can reduce these risks by:
- Verifying model and dataset origins.
- Recording versions and data lineage.
- Restricting who can modify training and retrieval sources.
- Checking the integrity of downloaded artifacts.
- Testing model behavior after significant updates.
CISA’s May 2025 guidance on securing AI data reinforces the importance of protecting data integrity throughout the AI lifecycle.
What are model extraction, inversion and evasion attacks?
These attacks target different properties of a machine learning system.
Model extraction attempts to reproduce a model’s behavior through repeated queries. It may threaten proprietary models exposed through APIs.
Model inversion attempts to recover information about training data from model outputs. Membership inference is a related privacy attack that seeks to determine whether a record was included in training.
Evasion uses manipulated inputs to make a model produce incorrect predictions.
Their relevance depends on the deployment model.
A company consuming a third-party LLM API usually has less control over model training and weights, but it remains responsible for application security.
An organization hosting or training its own models must additionally protect model artifacts, training data and inference infrastructure.
NIST’s adversarial machine learning taxonomy provides a useful reference for these distinctions.
What are LLM guardrails, and what are their limitations?
LLM guardrails are controls designed to detect, constrain or intervene in unsafe behavior across an AI application. They may operate on inputs, retrieved information, generated outputs, tool calls or runtime actions, using fixed rules, classifiers, additional language models or application-level checks.
Guardrails can identify suspicious prompts, enforce response formats, flag potentially sensitive content and help detect unsafe tool requests.
However, they are not a substitute for application security. Authorization, access restrictions and other critical security boundaries must be enforced independently of model instructions.
| Guardrails | Deterministic controls |
| Detect suspicious language | Enforce user permissions |
| Flag unsafe responses | Restrict API operations |
| Check generated content | Validate tool parameters |
| Can produce false positives or negatives | Apply explicit rules outside the model |
A guardrail might fail to recognize a sophisticated prompt injection. The application should still prevent the model from accessing unauthorized records or executing prohibited actions.
The model can propose an action, but the application must decide whether that action is permitted.
How do you limit the damage when an AI attack succeeds?
Preventing every prompt injection is not a realistic security assumption. A more resilient design limits what an attacker can achieve even when the model is manipulated.
This is the principle of containment.
Six controls that reduce the blast radius
| Control | What it prevents |
| Least-privilege access | The model accessing unnecessary data or tools |
| Independent authorization | Actions exceeding the user’s permissions |
| Output validation | Unsafe model output reaching downstream systems |
| Human approval | High-impact actions executing without review |
| Usage limits | Excessive requests consuming resources |
| Isolation and kill switches | A compromised workflow continuing unchecked |
Consider an AI assistant connected to a customer database.
An attacker may succeed in making the model request another customer’s information. But if the database API independently checks authorization, the request is rejected.
The model has been manipulated, yet the security boundary still holds.
OWASP LLM01:2026 (Prompt Injection), LLM03:2026 (Excessive Agency) and LLM10:2026 (Improper Output Handling) support this layered approach.
Before deployment, teams should answer three questions:
- What information can the model access?
- What actions can it perform through connected tools?
- What would happen if the model ignored every behavioral instruction?
The answers reveal the application’s potential blast radius.
How do you threat-model and red-team an AI application?
AI threat modeling identifies how an attacker might compromise a particular application. Red teaming tests whether those scenarios work in practice.
A practical five-step process is:
- Identify assets: Models, prompts, datasets, credentials and tools.
- Map entry points, trust boundaries and identities: Identify user input, uploaded files, retrieved documents and external APIs. Document which identities and components can access which data or tools, where information crosses trust boundaries and which sources or systems are trusted or untrusted.
- Define attack goals: Data theft, unauthorized actions, manipulation or disruption.
- Test controls: Attempt realistic attacks in a safe environment.
- Document findings: Record failures, containment results and residual risks.
Testing should include both malicious and legitimate requests.
A successful defense does not always mean the model refuses the malicious instruction. It may mean the model requests an unauthorized action but the application correctly blocks it.
Security teams should test again after changes to prompts, models, retrieval systems or tool permissions.
What changes when AI agents can use tools?
AI agents introduce additional risks because they can perform actions rather than only generate responses.
An agent may search databases, modify records, send messages or interact with external services.
Excessive agency remains an important risk, but it is only one part of the agentic threat model.
The OWASP Top 10 for Agentic Applications 2026 extends security guidance to threats specific to autonomous and multi-step workflows. These include agent goal hijacking (ASI01), tool misuse and exploitation (ASI02), identity and privilege abuse (ASI03), memory and context poisoning (ASI06), and insecure inter-agent communication (ASI07). Agentic systems can therefore be compromised through manipulated objectives, untrusted persistent context, excessive tool permissions or messages exchanged between agents, not just through unsafe individual model responses.
OWASP Gen AI Security Project+1
An agent should receive only the permissions required for its task. Sensitive operations should require independent authorization, and consequential actions may need human approval.
Persistent memory and messages from external tools or other agents should also be treated as potentially untrusted.
For a deeper discussion, see AI agents in business and agentic AI security threats.
Which AI security controls should be in place before launch?
An LLM application should not enter production without basic security evidence covering its data, model behavior, connected tools and operational response.
AI security checklist
| Control | What to verify |
| Threat model | Major attack paths and assets are documented |
| Access control | Tools and retrieval sources enforce permissions |
| Prompt injection testing | Direct and indirect attacks have been tested |
| Output handling | Generated content is validated before use |
| Supply chain | Models and dependencies have verified origins |
| Monitoring | Security events and abnormal usage are visible |
| Incident response | Owners and escalation procedures are defined |
| Containment | Sensitive tools or workflows can be disabled |
| Change management | Security tests run after significant updates |
Editorial checklist based on OWASP, NIST and NCSC guidance.
Each control should have a named owner.
The development team may own output validation, data owners may approve retrieval access, and security operations may manage incident response.
An effective AI-grade foundation combines these controls with reliable data governance, identity management and observability.
The goal is not to make an AI system impossible to manipulate. It is to ensure that manipulation does not automatically lead to unauthorized access or harmful actions.
How does AI security relate to AI safety and governance?
AI security, safety and governance address different but overlapping risks.
- AI security protects systems against adversarial attacks.
- AI safety addresses harmful or unreliable behavior, including harm that occurs without an attacker.
- AI governance defines accountability, policies and oversight.
- Legal compliance concerns applicable requirements such as GDPR and the EU AI Act.
NIST’s AI Risk Management Framework and its Generative AI Profile help organizations assess broader AI risks.
However, governance frameworks do not replace technical security controls. A compliant policy cannot compensate for an API that allows unauthorized data access.
FAQ
How serious is a prompt injection?
Its severity depends on what the AI system can access or do. A prompt injection may only alter an answer, or it may lead to confidential data exposure or unauthorized actions. Least privilege and independent authorization significantly reduce the potential damage.
What is the difference between MITRE ATLAS and MITRE ATT&CK?
MITRE ATT&CK describes adversarial tactics and techniques across conventional IT environments. MITRE ATLAS focuses on adversarial threats involving AI-enabled systems. Both support threat modeling, but ATLAS is more relevant to AI-specific attack scenarios.
Is MCP a security risk?
The Model Context Protocol (MCP) connects AI applications to external tools and data. Its use expands the attack surface when connected services have excessive permissions or return untrusted content. Authorization, tool isolation and monitoring are essential.
What is AI security posture management (AI-SPM)?
AI-SPM helps organizations identify AI assets, assess security configurations and monitor exposure. It improves visibility across models, applications and supporting infrastructure, but it does not replace application-level security testing.
Is AI safe to use?
AI systems can be deployed with acceptable risk when their capabilities, data exposure and permissions are appropriately controlled. The necessary safeguards depend on the use case, particularly whether the system can access sensitive information or execute consequential actions.
Sources
https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026
https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/README.md
https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026
https://genai.owasp.org/glossary/jailbreak-prompt
https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
https://csrc.nist.gov/pubs/ai/100/2/e2025/final
https://www.ncsc.gov.uk/guidance/ai-and-cyber-security-what-you-need-to-know