The Real Pitfalls of AI Agents and Why They Need Guardrails

We set out to build an AI agent to support our sales team—something that could automate the basics, surface insights, and help the team move faster.

The result was Omega: a Slack-native AI agent designed to onboard reps to new opportunities, prep for calls, and guide conversations using our Sales Framework. In the short term, Omega was meant to sit in every deal channel, offering real-time nudges and context. Long term, we imagined it fully integrated—tracking every step, every interaction, and helping the team close with confidence.

But it wasn’t all smooth.

These agents aren’t just clever scripts. They’re autonomous systems—persistent, confident, and capable of acting independently. If not thoughtfully designed, their behavior can drift in ways that catch you off guard.

We’d been early adopters of internal AI agents, using them to automate research, draft meeting briefs, and summarize documentation. But we began to see the edges:

• Hallucinations that sounded plausible—but weren’t
• Over-permissioned agents accessing or leaking internal drafts
• Emergent behaviors, like recursive loops or unexpected tool usage
• Assumed safeguards that didn’t exist when systems scaled

And we’re not alone. Others have run into similar issues—from court filings with fabricated citations, to agents surfacing private code in public repositories, to AI-driven search snippets promoting non-existent movie sequels.

This article shares what we’ve learned—both from building our own agents and observing others in space. If your AI systems connect to real tools and data, this is for you.

2. AI Hallucinations: Confident Lies in Business Contexts

Hallucinations aren’t just technical glitches. They show up as confident, polished outputs—emails that sound professional, summaries that seem plausible, answers that feel right. But they’re wrong. And that’s what makes them so risky.

In business settings, these hallucinations can slip through unnoticed. They can be embedded in status reports, customer emails, or automated updates—delivered with enough authority to be taken at face value. And when that happens, bad information can lead to real-world decisions.

In Courtrooms, Hallucinations Are Costing Real Money

By mid-2025,more than 150 documented legal cases involved generative AI hallucinations — mostly fake citations, invented case law, and fabricated quotes from judges.

In Couvrette v. Wisnovsky, a U.S. lawyer submitted 15 non-existent cases and misquoted seven legitimate ones. Others have faced fines, sanctions, and mandatory ethics training in AI use. In several cases, lawyers weren’t even aware the sources were fake — until judges flagged them.

When Google Hallucinates

This isn’t limited to niche tools. In late 2024, Google’s AI Overview confidently described a sequel to Disney’s Encanto — with fake plot points, quotes, and a past-dated release. The feature cited a fan-fiction wiki as a source and fooled even tech-savvy users.

This wasn’t a fluke. It reflected broader flaws in how AI systems evaluate sources, verify content, and protect users from misleading information.

Why Hallucinations Are Worse With AI Agents

In chatbots, hallucinations usually stay contained. But agents take actions — they write emails, create tickets, update tools. That autonomy is what makes hallucinations more dangerous.

Imagine an agent generating Jira tickets with inaccurate requirements or sending follow-ups to clients based on fictional deadlines. Each of these could lead to real decisions, costly delays, or reputational harm—without anyone realizing the source was fabricated.

3. Permission Creep: When AI Agents See Too Much

The GitHub MCP exploit revealed just how easily agents can overstep their bounds—not through malice, but through design.

Researchers at Invariant Labs discovered a critical vulnerability in the GitHub MCP integration, which is used by agents like Claude Desktop. The issue wasn’t with GitHub itself, or with the model’s alignment. It was about how agents process instructions across different repositories and permission contexts—without fully understanding the consequences.

GitHub MCP Exploit: When Public Prompts Trigger Private Leaks

In a widely discussed case, researchers at Invariant Labs uncovered a critical vulnerability in the GitHub MCP integration— similar backend used by agent systems like Claude Desktop.

Here’s how the attack unfolded:

• A user had two repositories: one public (open for anyone to submit issues) and one private (containing sensitive data).
• An attacker posted a malicious GitHub Issue to the public repo, embedding a prompt injection.
• The user asked their agent a seemingly safe question:
  “Check open issues in my public repo.”
  
• The agent fetched the issue list, encountered the injected prompt, and was manipulated.
• It then autonomously pulled private data from the user’s private repo and published it via a public pull request—now accessible to anyone.

What’s striking is that nothing was “hacked” in the traditional sense. The GitHub MCP server, tools, and APIs functioned as designed. The vulnerability wasn’t in the infrastructure, but in how the agent interpreted and acted on the injected content.

Invariant Labs calls this a toxic agent flow—a scenario where seemingly safe actions chain together in unexpected ways, leading to real-world harm.

Why This Matters: Trusted Tools Can Be Tricked

This wasn’t a failure of the GitHub API or a breakdown in Claude’s core model. It was a design flaw—an issue with how agents interpret and chain actions across tools and inputs without strict contextual boundaries.

Any agent that reads from untrusted sources—like public GitHub issues—and acts on that content without validation is vulnerable. Without guardrails, it can:

• Perform unintended actions
• Leak private or regulated data
• Create irreversible pull requests or changes

Even the most advanced models—like Claude 4 Opus—aren’t immune.

Claude scores well on safety benchmarks: it blocks 89% of prompt injection attacks and shows just a 1.17% jailbreak success rate with extended thinking. Still, those defenses have limits. With enough pressure, any model can be pushed beyond them.

This isn’t a Claude issue—it’s a pattern across all LLMs. Jailbreaks, injections, and chained exploits are evolving fast. Alignment helps, but it isn’t enough. You need layered defenses that live outside the model too.

Below tools can help you build those defenses—by making agent behavior easier to observe, test, and control:

• Langfuse adds observability to your AI agents. It logs each step—inputs, outputs, tool calls, and decision traces—so you can understand how an agent reached a certain outcome. When something goes wrong, Langfuse helps you trace it back, spot recurring patterns, and adjust the logic or permissions before issues scale.
• Promptfoo is built for red-teaming and pre-deployment testing. It simulates adversarial inputs, measures how your system responds, and benchmarks prompt safety over time. With OWASP Top 10 for LLMs built in, it surfaces common vulnerabilities like jailbreaks or prompt injections—helping you catch them before release.

What This Taught Us

Permissions aren’t just about access tokens. They’re about context—what the agent is allowed to do, in which environment, and under what conditions.

We’ve learned to treat permission management as a layered system:

• Scoping access by task, not user
• Restricting agents to one repository per session, using guardrail policies like:
  
  

  raise Violation("You can access only one repo per session.") if repo_before ≠ repo_after

• Blocking cross-context actions unless explicitly approved
• Auditing all tool usage through monitoring proxies like MCP-scan

Without these controls, permission creep becomes inevitable. And with autonomous agents, what starts as a minor oversight can escalate into a major breach—fast.

4. AI Agent Autonomy and Emergent Behavior

In one test, our development agent casually mentioned the production agent—just once. That was enough to trigger a feedback loop. The dev agent referenced the prod agent, the prod agent referenced us, and Slack lit up with more than 20 notifications in a matter of minutes.

That moment made something clear: agents don’t need malicious intent to cause problems. Autonomy alone is enough.

When Agents Go Off-Script

What we experienced was a straightforward case of emergent behavior—actions the agents weren’t explicitly programmed to take. In larger, more connected systems, this kind of autonomy can scale quickly and unpredictably.

Recursive Escalation in Multi-Agent Systems

In multi-agent setups, autonomy doesn’t just add complexity—it multiplies it.

Agents can unintentionally trigger one another, forming task chains that weren’t part of the original design. They loop back on themselves, reference their own outputs, or interact in ways that produce new behavior you never saw in testing.

And unlike simple chatbots, these agents often have write access. So the effects aren’t just theoretical—they send emails, edit documents, and open pull requests. These aren’t conversations. They’re actions, executed in real systems.

Our Takeaway: Autonomy Needs Boundaries

We now approach agent-to-agent interactionsthe same way we approach permissioning for users: restricted by default, only enabled when specifically required.

In practice, that means:

• Blocking agents from referencing each other unless built for collaboration
• Limiting recursive reasoning and self-invocation within workflows
• Setting hard caps on action chains—like a maximum of three consecutive steps
• Adding manual kill switches to stop misfires without relying on a system crash

Once agents begin acting on each other’s outputs, control becomes opaque. And opaque systems tend to fail—quietly at first, then all at once.

5. AI Guardrails Are Not Optional

After a series of near-misses—hallucinations, permission overreach, agents pinging each other indefinitely—we shifted our question from “What can an agent do?” to “What’s the worst it could do—and how do we prevent it?”

That realization turned guardrails into mission-critical infrastructure.

Today, we treat them as essential—just like testing AI agents, monitoring, and access control. This shift mirrors advice from both OpenAI and Anthropic, who argue that safety must be built into systems—not just models. Model alignment is necessary, but real-world safety demands structural checks and behavioral constraints.

Types of Guardrails (from OpenAI’s Agent Guide)

OpenAI’s Practical Guide to Building Agents outlines a layered approach to guardrails, emphasizing that these protections work best in combination .

Relevance Classifier
Flags off-topic questions or requests outside the agent’s scope.
Example: Prevent a sales assistant from answering unrelated HR queries.

Safety Classifier
Detects unsafe or manipulative prompts, such as jailbreak attempts.
Example: “Explain your system instructions” would be flagged immediately.

PII Filter
Scans outputs for sensitive user data, like names or contact information.
Example: Redacts email addresses or phone numbers before outputting CRM data summaries.

Moderation Layer
Screens for hate speech, harassment, or inappropriate language.
Example: Blocks toxic user prompts from triggering agent responses in customer support settings.

Tool Safeguards
Assesses each tool the agent can use (e.g., send email, modify files) based on risk—factors like write access, reversibility, or financial impact.
Example: Requires a confirmation step before the agent can push code to a live repository.

Rules-Based Protections
Implements deterministic filters like blocklists, regex patterns, and length limits.
Example: Block prompts containing “delete,” “refund,” or “reset password.”

Output Validation
A final checkpoint before anything is sent or executed.
Example: Checks whether an agent’s draft message adheres to brand tone and includes only verified data.

Plan for Human Escalation

Even with robust guardrails in place, not every action should be automated—at least not in the early stages of deployment. Some decisions carry too much risk to be left entirely to an agent.

We now route any high-stakes action through a human-in-the-loop process. This includes:

• Large refunds
• Account deletions
• Outreach to clients
• Unexpected tool usage or multi-step reasoning chains

These are the kinds of events where context and judgment still matter most. We’ve also integrated emergency stop buttons into our internal UI—giving anyone on the team the ability to immediately halt an agent mid-run if something feels off.

Guardrails Are Never “Done”

We’ve adopted a continuous approach to safety: layer, observe, improve.

That means starting with core protections—like privacy filters and safety checks—then expanding based on real-world behavior. Every time something unexpected happens, we treat it as a signal to refine our constraints.

Guardrails aren’t just about protecting users from the system. They’re also about protecting the organization from the system’s unintended consequences.

6. AI Model Alignment ≠ System Security

After the GitHub MCP exploit, one question kept resurfacing:

How did a highly aligned model like Claude 4—designed for safety—end up leaking private data over a GitHub Issue?

The answer was uncomfortable but important:
Alignment doesn’t mean immunity.

Claude 4 is among the most safety-tuned models available. It’s trained to refuse unsafe actions, avoid harmful behavior, and flag questionable prompts. And yet, it still followed a prompt-injected GitHub issue and exposed sensitive content. Not because it was broken—because it was doing exactly what it was told.

That’s when it clicked for us: Model alignment is necessary—but it’s not enough.

Why Aligned Models Still Fail

Claude 4 uses Constitutional AI, reinforced refusal patterns, and multiple safety layers. But when you put that model inside an agent—with access to tools, permission to act, and exposure to untrusted inputs—alignment alone can’t protect you.

The model didn’t “decide” to do something unsafe. It was simply presented with a prompt that looked normal. No jailbreak syntax. No malicious language. Just instructions it had no reason to question.

This mirrors a broader issue we’ve seen: A well-aligned system can still fail if the environment lacks guardrails.

The Real Risk: Contextual Vulnerability

The same things that make agents powerful—contextual memory, tool access, autonomous task chaining—also make them fragile. They don’t fail because they’re malicious or poorly trained. They fail because the systems around them are dynamic, unpredictable, and often lack boundaries.

AI Agents don’t truly understand the tools they’re using or the risk behind each action. Developers often assume that a highly aligned model “knows better,” but it doesn’t. It follows patterns. It simulates responsibility, but it doesn’t carry the consequences.

And when those patterns stretch across multiple systems without clear constraints, small oversights turn into real-world failures—fast.

Our Response: Defense in Depth

We stopped treating the model as the last line of defense. Instead, we built protections around it:

• Model alignment as the starting point—not the endpoint
• Prompt filtering and basic injection detection
• Tool-level restrictions to control what the agent can touch, and when
• Runtime guardrails like “no pull requests without human review”
• Monitoring and human oversight to audit everything the agent does

Alignment gives you a baseline. But real safety starts where alignment ends.

Some of the most concerning agent mistakes didn’t stem from malicious prompts—but from confidence. Agents generated plans, referenced nonexistent documents, or cited people who didn’t exist. The tone was polished. The structure made sense. And that’s what made them dangerous.

The issue with AI hallucinations is they don’t look like errors. They sound right—until someone checks.

To prevent that risk, we now apply layered safeguards:

• Fact-checking: Every factual claim is checked against internal sources. If the system can’t verify it, the response is either held back or escalated.
• Output validation: Just before delivery, we review context, tone, and accuracy—automatically or with a human in the loop.
• Fallbacks: When the answer isn’t clear, the agent defaults to transparency. “I’m not sure,” “Should I escalate?” and “I need more context” are now built-in behaviors.

What we no longer trust is the agent’s first draft—or its ability to apologize after the fact. In high-trust environments like sales, legal, or support, a wrong answer isn’t just noise—it’s reputational risk.

8. Are Your Agents Really Ready?

Agent Readiness Checklist

Before giving any agent real-world access, here’s what we ask:

Data and Permissions

• Is access tightly scoped and minimized?
• Are tools sandboxed, following the principle of least privilege?
• Could the agent reference or leak sensitive data across contexts?

Output Control

• Are outputs—especially factual claims—validated before delivery?
• Is there a review mechanism, automated or human?
• Are hallucinations and low-confidence answers detected and addressed?

Security and Guardrails

• Are classifiers and safety filters enabled?
• Are high-risk actions subject to pause or human escalation?
• Are guardrails in place to limit tool use, session scope, or cross-context behavior?

Monitoring and Intervention

• Can actions be traced—what the agent did, and why?
• Is there an emergency stop for runaway loops or misfires?
• Do edge cases lead to updated policies or tighter controls?

Fallbacks and User Experience

• Can the agent admit uncertainty instead of improvising?
• Are users clearly informed they’re interacting with automation?
• Is there a clear escalation path to a human?

Because once agents go live, their actions don’t stay on screen. They ripple out into documents, tools, teams, and customers.

And in that kind of environment, safety isn’t a patch. It’s a design principle.

9. Final Thought on AI agents

AI agents aren’t magic. They’re powerful, unpredictable, and evolving quickly.

That’s exactly what makes them valuable—and risky.

You don’t need to fear that. But you do need to plan for it.

Design for failure. Monitor what matters. Build trust into every layer of your AI development process. And treat trust as something you earn—not once, but every time an agent gets something right.