When people talk about data leaks and AI, they usually mean one of two different problems:
- Data going in: staff paste confidential or personal data into AI tools, where it may be stored, used for training or exposed.
- Data coming out: an AI chatbot reveals information it shouldn’t, to the wrong user or to an attacker.
Both are common, and both are preventable.
Leak path 1: Staff pasting sensitive data into AI tools
This is the most common AI data leak, and it’s usually well-intentioned: someone wants a faster summary, a better email or help with code. What gets pasted:
- Customer and employee personal data
- Contracts, client files and legal documents
- Source code, configuration files and logs, which often contain API keys or passwords
- Financial results, deal details and strategy documents
Why it matters: depending on the tool and plan, inputs may be stored, reviewed by the provider, used to train future models, or kept in chat history that others with access to the account can see. It can also breach GDPR, client contracts and confidentiality duties.
How to prevent it:
- Give staff an approved, business-grade AI tool with contractual data protection, training opt-out and admin controls. Bans without alternatives push people to personal accounts.
- Publish clear data rules in an AI acceptable use policy.
- Remove sensitive details before pasting: replace names, account numbers and secrets with placeholders.
- Use data loss prevention controls where available: browser or endpoint tools that warn or block when sensitive data is pasted into AI sites.
- Train people. It’s also part of the AI literacy duty under the EU AI Act.
Leak path 2: Chatbots revealing data
When you build or deploy a chatbot connected to your data, it can leak information through:
- Over-broad access. The bot can read documents or records the current user isn’t allowed to see, and a well-phrased question gets them out.
- Prompt injection. Attackers trick the bot into revealing its context, other users’ data or its system prompt.
- Secrets in the system prompt. API keys or internal rules placed in the prompt can be extracted.
- Shared memory or history. Conversation data bleeding between users or sessions.
- Training on customer data. Fine-tuning on sensitive data that the model can later reproduce.
How to prevent it:
- Enforce permissions outside the model. The chatbot should only retrieve data the signed-in user is already allowed to see. Never rely on the prompt to say “don’t show this”.
- Minimise data. Connect only what the bot needs for its job.
- Keep secrets out of prompts, and assume anything in the prompt can be read.
- Filter outputs for personal data patterns and secrets before showing responses.
- Separate users and tenants in conversation history and retrieval indexes.
- Test it. Red-team the chatbot specifically for data extraction before launch.
- Log and monitor for unusual queries and large data outputs.
If a leak happens
- Contain it: revoke exposed credentials immediately and disable the affected feature if needed.
- Find out what data was exposed, to whom, and for how long.
- Check your obligations. A personal data breach may need to be reported to your data protection authority, under GDPR within 72 hours of becoming aware of it, and to affected people.
- Fix the root cause, and update training and controls.
The short version
Stop sensitive data going in with approved tools, clear rules and data loss prevention. Stop it coming out with permission-aware design, data minimisation and testing. Most AI data leaks are preventable with controls you can put in place this month.
Official sources
- GDPR, Regulation (EU) 2016/679: the EU’s data protection law
- European Data Protection Board: EU regulators’ guidelines and opinions, including on AI models
- ICO guidance on AI and data protection: the UK regulator’s guidance under the UK GDPR
- OWASP Top 10 for LLM Applications: the OWASP GenAI Security Project’s list of the main LLM risks
More primary sources, with a short note on each, are on our Sources page.
Last reviewed October 2026. This article is general information, not legal advice.