AI security: prompt injection, data leakage and model threats, and how to limit the damage

AI security: prompt injection, data leakage and model threats, and how to limit the damage

AI security protects the AI systems a company builds or deploys from attacks such as prompt injection, data leakage, poisoning and model theft. This guide maps ten OWASP risks onto five attack layers. It shows which controls limit the damage when an attack succeeds.

What is AI security, and how is it different from AI for cybersecurity?

AI security protects models, data and AI-powered applications against attacks and misuse. It differs from AI for cybersecurity, which uses artificial intelligence to detect threats, analyze suspicious activity or support security teams.

There are three related concepts:

ConceptMeaning
Security of AIProtecting AI models, applications, data and connected tools
AI for cybersecurityUsing AI to improve security operations
AI-enabled attacksUsing AI to assist phishing, fraud or other malicious activity

This guide focuses on security of AI, particularly enterprise applications built around large language models.

These systems introduce risks that conventional application security does not fully address. An attacker may manipulate retrieved documents, influence model behavior or exploit the permissions granted to an AI assistant.

What does the attack surface of an AI system look like?

For this guide, we group the AI application attack surface into five practical layers: prompts, retrieved data, models and dependencies, outputs and tools, and infrastructure. This is a threat-modeling structure used to organize the risks and controls discussed here, not a standardized five-layer AI security taxonomy. Each layer creates opportunities for attackers to influence behavior, access information or trigger unauthorized actions.

Five layers of AI security

LayerTypical threatMain defensive approach
Input and promptsPrompt injection and jailbreakingInput checks and restricted capabilities
Retrieved dataMalicious documents and unauthorized retrievalSource validation and permission-aware retrieval
Model and supply chainPoisoning and compromised dependenciesProvenance checks and artifact verification
Output and toolsUnsafe generated commands and excessive agencyOutput validation and least privilege
InfrastructureCredential theft and resource abuseIdentity controls, rate limits and monitoring

The main difference from conventional software is that LLMs process instructions and external information through a shared language context.

A document retrieved for summarization can contain malicious instructions. If the model follows them, the application may behave differently without any change to its code.

This is why securing the model alone is insufficient. The surrounding application must independently enforce its security boundaries.

What are the main AI security risks according to OWASP?

The OWASP Top 10 for LLM Applications 2026, published in August 2026, identifies ten major risks affecting applications built around large language models. This edition updates the ordering and scope of the previous list, including replacing System Prompt Leakage with the broader category Hidden Context Exposure. The risks provide a starting point for threat modeling and prioritizing security controls.

OWASP riskWhat can go wrongKey control
LLM01:2026 Prompt InjectionMalicious instructions redirect the modelRestrict permissions and tool access
LLM02:2026 Sensitive Information DisclosureConfidential data appears in responsesData minimization and authorization
LLM03:2026 Excessive AgencyAI performs unauthorized actionsLeast privilege and human approval
LLM04:2026 Supply ChainCompromised models or dependencies enter the systemVerify origins and versions
LLM05:2026 Data and Model PoisoningManipulated data changes model behaviorValidate datasets and track lineage
LLM06:2026 Unbounded ConsumptionExcessive usage exhausts resources or budgetsQuotas and rate limits
LLM07:2026 MisinformationIncorrect answers influence decisionsVerify consequential outputs
LLM08:2026 Hidden Context ExposureNon-user-facing instructions, tool schemas or operational context are exposedKeep secrets out of model context and enforce authorization independently
LLM09:2026 Vector and Embedding WeaknessesRetrieval exposes unsuitable or restricted contentEnforce retrieval permissions
LLM10:2026 Improper Output HandlingGenerated output causes downstream vulnerabilitiesValidate and safely process output

Controls are an editorial summary based on OWASP’s 2026 risk catalogue.

Hidden Context Exposure is broader than disclosure of a system prompt. It includes hidden developer instructions, retrieved policy context, tool and function schemas, and other operational information supplied to the model but not intended for users. Applications should assume that such context may be discoverable. Credentials must not be stored in it, and authorization, privilege separation and other security boundaries must not depend on keeping it secret.

GitHub

OWASP describes common application risks, while MITRE ATLAS documents adversarial techniques targeting AI systems. NIST AI 100-2 E2025 provides a taxonomy of adversarial machine learning threats.

Together, these resources help teams identify vulnerabilities, understand attack methods and develop realistic security tests.

How does prompt injection work, and can it be prevented?

Prompt injection occurs when an attacker introduces instructions that redirect an AI application’s intended behavior.

It can take several forms:

  • Direct prompt injection: Malicious instructions are submitted directly to the assistant.
  • Indirect prompt injection: Instructions are hidden in documents, webpages, emails or tool responses that the assistant processes.
  • Multimodal injection: Adversarial instructions are embedded in images or other supported media.

For example, an assistant may retrieve a webpage while researching a legitimate question. The page contains instructions telling the model to ignore its task and reveal internal information.

The attacker wants the model to treat external content as authoritative instructions.

Prompt injection vs jailbreaking

Prompt injection and jailbreaking are related but distinct attack concepts.

Prompt injection introduces attacker-controlled instructions that attempt to override the application’s or developer’s intended behavior. It can manipulate an application without necessarily asking the model to generate prohibited content.

Jailbreaking specifically attempts to bypass a model’s trained safety restrictions, causing it to produce outputs it would normally refuse. The two techniques can overlap or be combined, but jailbreaking is not necessarily a subtype of prompt injection.

OWASP notes that foolproof prompt injection prevention remains uncertain. Filters and prompt instructions reduce risk, but they cannot guarantee that every malicious input will be rejected.

The stronger approach combines detection with controls the model cannot override, including authorization checks, restricted tools and approval for sensitive actions.

How does AI data leakage happen?

AI data leakage occurs when sensitive information becomes accessible through model outputs, retrieved context, training data or connected tools.

Four common routes are:

Leak routeExamplePrevention
Training dataSensitive examples appear in generated responsesData minimization and privacy controls
Retrieved documentsAssistant accesses files the user cannot openPermission-aware retrieval
Hidden application contextInternal instructions, credentials or sensitive operational details become exposedKeep secrets outside model context and enforce security boundaries independently
Tool outputsConfidential information reaches external servicesRestrict outbound actions

Why RAG permissions matter

Retrieval-augmented generation (RAG) allows an LLM to use external knowledge. However, a retrieval system may access more documents than the person asking the question.

If the application retrieves confidential information before checking permissions, the model may expose it in its answer.

The solution is to enforce authorization before information enters the model’s context.

Document-level permissions, access isolation and source validation are essential to secure RAG applications.

This article addresses leakage from systems an organization builds or deploys. Employees sharing information through unauthorized AI tools represent a related but separate shadow AI risk.

How do data poisoning and supply chain attacks affect AI?

Data poisoning manipulates information used to train, fine-tune or support an AI system. The objective may be to introduce false associations, alter predictions or create hidden behaviors.

For example, an attacker who can modify an internal knowledge-base document may insert false business instructions. When the assistant retrieves that document, it may repeat the manipulated information.

Supply chain attacks target external dependencies, including model checkpoints, datasets, libraries and packages.

Organizations can reduce these risks by:

  • Verifying model and dataset origins.
  • Recording versions and data lineage.
  • Restricting who can modify training and retrieval sources.
  • Checking the integrity of downloaded artifacts.
  • Testing model behavior after significant updates.

CISA’s May 2025 guidance on securing AI data reinforces the importance of protecting data integrity throughout the AI lifecycle.

What are model extraction, inversion and evasion attacks?

These attacks target different properties of a machine learning system.

Model extraction attempts to reproduce a model’s behavior through repeated queries. It may threaten proprietary models exposed through APIs.

Model inversion attempts to recover information about training data from model outputs. Membership inference is a related privacy attack that seeks to determine whether a record was included in training.

Evasion uses manipulated inputs to make a model produce incorrect predictions.

Their relevance depends on the deployment model.

A company consuming a third-party LLM API usually has less control over model training and weights, but it remains responsible for application security.

An organization hosting or training its own models must additionally protect model artifacts, training data and inference infrastructure.

NIST’s adversarial machine learning taxonomy provides a useful reference for these distinctions.

What are LLM guardrails, and what are their limitations?

LLM guardrails are controls designed to detect, constrain or intervene in unsafe behavior across an AI application. They may operate on inputs, retrieved information, generated outputs, tool calls or runtime actions, using fixed rules, classifiers, additional language models or application-level checks.

Guardrails can identify suspicious prompts, enforce response formats, flag potentially sensitive content and help detect unsafe tool requests.

However, they are not a substitute for application security. Authorization, access restrictions and other critical security boundaries must be enforced independently of model instructions.

GuardrailsDeterministic controls
Detect suspicious languageEnforce user permissions
Flag unsafe responsesRestrict API operations
Check generated contentValidate tool parameters
Can produce false positives or negativesApply explicit rules outside the model

A guardrail might fail to recognize a sophisticated prompt injection. The application should still prevent the model from accessing unauthorized records or executing prohibited actions.

The model can propose an action, but the application must decide whether that action is permitted.

How do you limit the damage when an AI attack succeeds?

Preventing every prompt injection is not a realistic security assumption. A more resilient design limits what an attacker can achieve even when the model is manipulated.

This is the principle of containment.

Six controls that reduce the blast radius

ControlWhat it prevents
Least-privilege accessThe model accessing unnecessary data or tools
Independent authorizationActions exceeding the user’s permissions
Output validationUnsafe model output reaching downstream systems
Human approvalHigh-impact actions executing without review
Usage limitsExcessive requests consuming resources
Isolation and kill switchesA compromised workflow continuing unchecked

Consider an AI assistant connected to a customer database.

An attacker may succeed in making the model request another customer’s information. But if the database API independently checks authorization, the request is rejected.

The model has been manipulated, yet the security boundary still holds.

OWASP LLM01:2026 (Prompt Injection), LLM03:2026 (Excessive Agency) and LLM10:2026 (Improper Output Handling) support this layered approach.

Before deployment, teams should answer three questions:

  1. What information can the model access?
  2. What actions can it perform through connected tools?
  3. What would happen if the model ignored every behavioral instruction?

The answers reveal the application’s potential blast radius.

How do you threat-model and red-team an AI application?

AI threat modeling identifies how an attacker might compromise a particular application. Red teaming tests whether those scenarios work in practice.

A practical five-step process is:

  1. Identify assets: Models, prompts, datasets, credentials and tools.
  2. Map entry points, trust boundaries and identities: Identify user input, uploaded files, retrieved documents and external APIs. Document which identities and components can access which data or tools, where information crosses trust boundaries and which sources or systems are trusted or untrusted.
  3. Define attack goals: Data theft, unauthorized actions, manipulation or disruption.
  4. Test controls: Attempt realistic attacks in a safe environment.
  5. Document findings: Record failures, containment results and residual risks.

Testing should include both malicious and legitimate requests.

A successful defense does not always mean the model refuses the malicious instruction. It may mean the model requests an unauthorized action but the application correctly blocks it.

Security teams should test again after changes to prompts, models, retrieval systems or tool permissions.

What changes when AI agents can use tools?

AI agents introduce additional risks because they can perform actions rather than only generate responses.

An agent may search databases, modify records, send messages or interact with external services.

Excessive agency remains an important risk, but it is only one part of the agentic threat model.

The OWASP Top 10 for Agentic Applications 2026 extends security guidance to threats specific to autonomous and multi-step workflows. These include agent goal hijacking (ASI01), tool misuse and exploitation (ASI02), identity and privilege abuse (ASI03), memory and context poisoning (ASI06), and insecure inter-agent communication (ASI07). Agentic systems can therefore be compromised through manipulated objectives, untrusted persistent context, excessive tool permissions or messages exchanged between agents, not just through unsafe individual model responses.

OWASP Gen AI Security Project+1

An agent should receive only the permissions required for its task. Sensitive operations should require independent authorization, and consequential actions may need human approval.

Persistent memory and messages from external tools or other agents should also be treated as potentially untrusted.

For a deeper discussion, see AI agents in business and agentic AI security threats.

Which AI security controls should be in place before launch?

An LLM application should not enter production without basic security evidence covering its data, model behavior, connected tools and operational response.

AI security checklist

ControlWhat to verify
Threat modelMajor attack paths and assets are documented
Access controlTools and retrieval sources enforce permissions
Prompt injection testingDirect and indirect attacks have been tested
Output handlingGenerated content is validated before use
Supply chainModels and dependencies have verified origins
MonitoringSecurity events and abnormal usage are visible
Incident responseOwners and escalation procedures are defined
ContainmentSensitive tools or workflows can be disabled
Change managementSecurity tests run after significant updates

Editorial checklist based on OWASP, NIST and NCSC guidance.

Each control should have a named owner.

The development team may own output validation, data owners may approve retrieval access, and security operations may manage incident response.

An effective AI-grade foundation combines these controls with reliable data governance, identity management and observability.

The goal is not to make an AI system impossible to manipulate. It is to ensure that manipulation does not automatically lead to unauthorized access or harmful actions.

How does AI security relate to AI safety and governance?

AI security, safety and governance address different but overlapping risks.

  • AI security protects systems against adversarial attacks.
  • AI safety addresses harmful or unreliable behavior, including harm that occurs without an attacker.
  • AI governance defines accountability, policies and oversight.
  • Legal compliance concerns applicable requirements such as GDPR and the EU AI Act.

NIST’s AI Risk Management Framework and its Generative AI Profile help organizations assess broader AI risks.

However, governance frameworks do not replace technical security controls. A compliant policy cannot compensate for an API that allows unauthorized data access.

FAQ

How serious is a prompt injection?

Its severity depends on what the AI system can access or do. A prompt injection may only alter an answer, or it may lead to confidential data exposure or unauthorized actions. Least privilege and independent authorization significantly reduce the potential damage.

What is the difference between MITRE ATLAS and MITRE ATT&CK?

MITRE ATT&CK describes adversarial tactics and techniques across conventional IT environments. MITRE ATLAS focuses on adversarial threats involving AI-enabled systems. Both support threat modeling, but ATLAS is more relevant to AI-specific attack scenarios.

Is MCP a security risk?

The Model Context Protocol (MCP) connects AI applications to external tools and data. Its use expands the attack surface when connected services have excessive permissions or return untrusted content. Authorization, tool isolation and monitoring are essential.

What is AI security posture management (AI-SPM)?

AI-SPM helps organizations identify AI assets, assess security configurations and monitor exposure. It improves visibility across models, applications and supporting infrastructure, but it does not replace application-level security testing.

Is AI safe to use?

AI systems can be deployed with acceptable risk when their capabilities, data exposure and permissions are appropriately controlled. The necessary safeguards depend on the use case, particularly whether the system can access sensitive information or execute consequential actions.

Sources

https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026

https://github.com/GenAI-Security-Project/GenAI-LLM-Top10/blob/main/2026/README.md

https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026

https://genai.owasp.org/glossary/jailbreak-prompt

https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html

https://csrc.nist.gov/pubs/ai/100/2/e2025/final

https://atlas.mitre.org

https://www.ncsc.gov.uk/guidance/ai-and-cyber-security-what-you-need-to-know

https://www.cisa.gov/news-events/alerts/2025/05/22/new-best-practices-guide-securing-ai-data-released

https://www.nist.gov/itl/ai-risk-management-framework

Translate »