--- title: "AI Agent Safety" description: "Your LLMs will hallucinate. Your safety flags will fail. Caro provides pattern-based command validation that catches dangerous commands—whether from users or confused AI agents. Deterministic protection for enterprise AI deployments." url: "https://caro.sh/ai-agent-safety" markdown_url: "https://caro.sh/ai-agent-safety.md" lang: "en" type: "website" image: "https://caro.sh/og/ai-agent-safety.png" --- AI Agent Safety Layer # AI Agents Run Dangerous Commands. Caro Catches Them. Your LLMs will hallucinate. Your flags will fail. Caro doesn't rely on either. Pattern-based validation that catches destructive commands—whether they came from a user or a confused AI. Claude Code deleted entire home directory despite --dangerously-skip-permissions Gemini CLI hallucinated file paths and deleted files that didn't exist [Protect Your Systems](https://caro.sh/ai-agent-safety#download) [Calculate Your Risk](https://caro.sh/ai-agent-safety#risk-calculator) Hallucination Resistant Pattern-Based Deterministic Open Source ## Why Flags Aren't Enough Real incidents from 2025 where safety measures failed Claude Code `--dangerously-skip-permissions` What happened: AI ran rm -rf \~/ anyway, deleting the user's entire home directory Why it failed: The flag controls permission prompts, not command validation. The AI didn't need permission—it just executed. — HN Discussion, Dec 2025 Gemini CLI `Built-in safety checks` What happened: AI hallucinated file paths that didn't exist, then confidently deleted real files while 'cleaning up' Why it failed: LLMs are stochastic. They make up data with high confidence. Safety checks can't catch hallucinations. — HN Discussion, Jul 2025 ### Flag-Based Safety vs Pattern-Based Safety Flag-Based (Others) - Trust the AI to remember the flag - Hope the flag covers all edge cases - Can't catch hallucinated commands - Probabilistic protection - Fails silently when AI gets creative Pattern-Based (Caro) - Validates every command, regardless of source - 52+ dangerous patterns compiled in - Catches hallucinations—pattern is pattern - Deterministic protection - Explicit warnings before execution 💡 **The key difference:** Flags ask "did the user consent?" Caro asks "is this command dangerous?" The AI doesn't need consent to run rm -rf \~/. It needs to be stopped. ## Calculate Your AI Risk The math of probabilistic failures at enterprise scale Developers using AI tools 500 AI commands per developer per day 50 LLM accuracy rate (%) 99.0% Daily AI commands 25,000 × Failure rate 1.0% \= Potential dangerous commands 250/day Without Caro 250 potentially dangerous commands daily that could slip through One unlucky user getting `rm -rf ~/` impacts your entire system With Caro 0 dangerous patterns executed without warning Deterministic validation—no dice rolls, no "demo gods" Hundreds of developers × probabilistic AI = guaranteed failures. Caro is your insurance policy. [Get Started Free](https://caro.sh/ai-agent-safety#download) ## This Happened. It Will Happen Again. Documented incidents where AI tools caused real damage Claude Code December 2025 ### Deleted user's entire home directory Despite using --dangerously-skip-permissions flag, Claude Code executed rm -rf \~/ when asked to 'clean up temp files'. The flag only controlled permission prompts—it didn't validate the actual commands. Impact: Complete loss of home directory including code, documents, and configuration files Key lessons: - Permission flags don't validate commands - AI can misinterpret 'cleanup' requests destructively - No backup strategy can help if deletion happens instantly [Read the HN discussion →](https://news.ycombinator.com/item?id=42768072) Gemini CLI July 2025 ### Hallucinated file paths and deleted real files Gemini CLI confidently fabricated file paths that didn't exist, then 'cleaned up' by deleting actual files in similar locations. The AI showed no uncertainty about its hallucinated information. Impact: Deletion of project files and configuration that the user never intended to modify Key lessons: - LLMs hallucinate with high confidence - Safety checks can't catch made-up paths - Stochastic systems will eventually fail [Read the HN discussion →](https://news.ycombinator.com/item?id=44651485) 📅 **Two major incidents in 2025 alone.** As AI coding tools become more popular, these incidents will increase. The question isn't *if* it will happen to your team—it's *when*. Claude Code ### Deleted entire home directory \--dangerously-skip-permissions flag didn't help — HN, Dec 2025 Gemini CLI ### Hallucinated file paths and deleted files AI confidently made up paths that didn't exist — HN, Jul 2025 **These aren't edge cases. LLMs are probabilistic systems—failures are inevitable at scale.** ## AI Agent Deployment Best Practices Defense in depth for AI-powered shell tools 👤 ### Run as Unprivileged User Never run AI tools with sudo or as root. Create a dedicated user with minimal permissions. `useradd --no-create-home --shell /bin/false caro-agent` Caro: Caro blocks sudo/su commands and warns when running as root 📁 ### Sandbox to Specific Directories Confine AI agents to specific directories. Protect /home, /etc, and system paths. `# Caro warns on operations outside working directory` Caro: Detects and warns on /home, \~, /etc, /usr operations 🐳 ### Container Isolation Run AI tools in containers with no access to important data or host filesystem. `docker run --rm -v $(pwd):/workspace caro-sandbox` Caro: Works in containers with zero host access needed 🎲 ### Assume Hallucinations Will Happen LLMs are probabilistic. Even with 99% accuracy, 1 in 100 commands could be dangerous. `# AI may fabricate paths: rm -rf /imaginary/but/destructive` Caro: Pattern matching catches dangerous commands regardless of source ### Defense in Depth: Don't Rely on Flags Alone 1 Unprivileged User (no sudo) 2 Directory Sandboxing 3 Container Isolation 4 Caro Pre-Execution Validation Each layer catches what the others miss. Caro is your last line of defense—not your only one. ### The Math of AI Risk at Enterprise Scale 1,000 developers × 100 AI commands/day \= 100,000 potential dangerous commands/day Even at 99.9% AI accuracy, that's **100 potentially dangerous commands daily**. One bad hallucination without Caro = catastrophe. ## Common Concerns Real questions from skeptical engineers (we get it) Wait, Caro runs locally. How does it stay updated with new dangerous patterns? + Caro's safety patterns are baked into the binary—no network needed. When you update Caro (cargo install caro --force), you get the latest patterns. The core dangerous commands (rm -rf /, fork bombs, disk wipers) don't change. We also accept pattern contributions via GitHub. Will this slow down my incident response? + No. Caro adds <100ms to command generation. The safety check is instant (pattern matching, not AI inference). In a real incident, that's 100ms that might save you from making things 10x worse. The validation is synchronous—you see the warning immediately. How does it know my specific system setup (BSD vs GNU, etc.)? + Caro detects your OS and shell at runtime. On macOS, it knows you're using BSD tools. On Linux, it adjusts for GNU syntax. It reads your $SHELL and adjusts accordingly. No configuration needed—it just works. What if I actually NEED to run a dangerous command? + Caro warns, it doesn't jail. When you see a warning, you can still proceed—we just make sure you're doing it intentionally. For truly destructive commands (rm -rf /), you'll need to confirm. This is your seatbelt, not a straitjacket. Is this just another AI wrapper that sends my commands to the cloud? + No. Caro runs 100% locally. Your commands, file paths, server names, and directory structures never leave your machine. The inference happens on your hardware. We have zero telemetry. Check the source code—it's AGPL-3.0 licensed. Why should I trust AI-generated shell commands at all? + You shouldn't trust them blindly—that's the point. Caro generates commands AND validates them before you run them. It's not 'trust the AI'—it's 'trust the pattern-based safety layer that catches what the AI might get wrong.' The validation is deterministic, not probabilistic. Still skeptical? Good—you should be. [Read the source code →](https://github.com/wildcard/caro) ## Try Caro in 30 Seconds No account. No API key. No data collection. Just safer shell commands. `bash <(curl --proto '=https' --tlsv1.2 -sSfL https://setup.caro.sh)` Then run: `caro "find files modified in the last 7 days"` Installs to \~/.cargo/bin Single binary, no dependencies Uninstall anytime: cargo uninstall caro Prefer to build from source? [See all installation options →](https://github.com/wildcard/caro#installation)