Stop AI Agents from Deleting Production Data

Imagine you're in a DevOps interview and the interviewer asks:
"Your AI agent is about to delete a production volume. How would you stop it?"
A common answer is: "I'll add an instruction to its prompt telling it not to delete anything."
That's not enough.
A prompt is a request, not a security boundary. An AI agent can ignore instructions, misunderstand its task, or make a dangerous decision while trying to solve an unrelated problem.
Let's look at a real incident that shows why this matters.

---

💥 The PocketOS Incident: A Production Database Deleted in 9 Seconds​

In April 2026, an AI coding agent running Cursor and Claude Opus 4.6 was working on a routine task in PocketOS's staging environment.
It encountered a credential mismatch and decided to fix the problem by deleting a Railway storage volume.
The problem? It found an API token with account-wide permissions and used it to delete the production volume instead.
The deletion took just 9 seconds.
Worse, the volume-level backups were also affected because they were stored alongside the volume. PocketOS's latest recoverable backup was three months old.
Railway eventually helped recover the data, but the incident exposed a serious problem: the agent had access to infrastructure far beyond what its task required.
Sources: Decrypt | The Incident Details

---

🔐 How Would You Prevent This? Use 4 Layers of Protection​

The right answer isn't a better prompt. It's multiple independent security controls.

1️⃣ Least Privilege: Limit What the Agent Can Access​

Ask yourself:
Can the agent's API token access production in the first place?

• Use separate credentials for development, staging, and production.
• Give each token only the permissions required for its task.
• Never give an AI agent account-wide administrator access by default.
• Avoid exposing unnecessary production credentials to development environments.

Remember: If the agent doesn't have permission to delete a production volume, it can't delete it through that credential, regardless of what it decides.

2️⃣ Human Approval: Block Dangerous Actions Outside the AI Model​

Should the AI agent be allowed to approve its own destructive actions?
Absolutely not.
High-risk operations such as DELETE, DROP, and production resource removal should pass through an independent approval mechanism.
For example, you can build a circuit breaker that:

• Detects potentially destructive operations.
• Blocks execution before the request reaches the infrastructure.
• Requires explicit human approval.
• Records the decision for auditing.

This is the approach my team and I worked on at MarsadOps, which won first place at DevOpsDays Cairo Hackathon.
The key idea: The AI agent must not control the mechanism responsible for approving its own dangerous actions.

3️⃣ Infrastructure Protection: Add Safety Controls at the Platform Level​

Don't rely entirely on the agent or your application.
Use the infrastructure platform's built-in protection mechanisms.
Kubernetes:
Check the PersistentVolume reclaim policy.
If the policy is Delete, deleting the associated PersistentVolumeClaim can also delete the PersistentVolume.
For important data, consider using Retain instead.
Example:

apiVersion: v1
kind: PersistentVolume
metadata:
name: production-volume
spec:
persistentVolumeReclaimPolicy: Retain

AWS RDS:
Enable deletion protection for production databases.
This prevents a database instance from being deleted until deletion protection is disabled.
These controls provide an additional safety layer even when an agent makes a mistake.

4️⃣ Disaster Recovery: Make Sure You Can Actually Restore Your Data​

A backup is useless if the same operation that deletes your production data can also delete its backups.
Follow these practices:

• Store backups in a separate account or environment.
• Use independent credentials that the AI agent cannot access.
• Enable immutable backups where appropriate.
• Maintain multiple recovery points.
• Test restoration regularly.

Important: Having backups is not the same as having a tested recovery plan.

---

🎯 What Is the Interviewer Really Looking For?​

The interviewer isn't looking for a list of tools.
They want to understand how you think about Blast Radius.
In simple terms, blast radius means how much damage an incident can cause and how far its impact can spread.
A strong answer demonstrates that you've considered:

Security Layer	Main Question
Permissions	Can the agent access production?
Human approval	Can it execute destructive actions without approval?
Infrastructure	Can the platform prevent accidental deletion?
Recovery	Can we restore data if everything else fails?

The goal is to make sure that one AI mistake doesn't become a company-wide disaster.

---

💡 Final Takeaway​

Never treat an AI agent's prompt as your security system.
Use least-privilege credentials, independent approval mechanisms, infrastructure-level protections, and isolated backups.
A secure AI agent isn't one that promises not to make mistakes. It's one that cannot cause catastrophic damage when it does.
💬 One question for DevOps engineers:
What can your current AI agent actually delete with the API token it has access to?
Share your experience in the comments, and follow for more practical DevOps, AI, and cybersecurity discussions.