Building AI Agents in Production: What MCP Rejections, .env Leaks, and 70-Line Loops Taught Me

Most AI agent architectures fail not because the model is wrong, but because the engineering around it is naive. I learned this the hard way when my first production agent loop crashed the billing service after a Model Context Protocol rejection cascaded into an environment variable leak and a 70-line execution trace that never terminated. This article isn't about prompt engineering—it's about the plumbing that separates toy demos from systems that run in production without waking up your on-call engineer at 3 AM.

The Death of the Single-Hop Agent

When building AI agents for production, the first mistake is assuming a single LLM call plus a tool invocation is sufficient. Real systems require stateful, multi-step reasoning with explicit failure boundaries. The Model Context Protocol (MCP) was designed to solve exactly this interoperability problem, but in practice, MCP rejections expose deeper architectural flaws in how we think about agent reliability.

An MCP server rejection isn't just a failed tool call—it's a signal that the agent has entered an unsupported context. In my experience, these rejections happen most frequently when:

• The LLM attempts to call tools in an order the MCP server doesn't support
• Context windows exceed the MCP server's buffer capacity
• Authentication tokens expire mid-session without graceful degradation
• Rate limits are hit without backoff logic in the agent controller

The critical realization: MCP rejections should be treated as first-class events, not exceptions. Every production agent architecture needs a rejection handler that transforms MCP failures into recoverable state transitions rather than silent crashes.

interface MCPRejectionHandler {
  handle(rejection: MCPError): Promise<AgentStateTransition>;
}

class ResilientAgentController implements MCPRejectionHandler {
  async handle(rejection: MCPError): Promise<AgentStateTransition> {
    switch (rejection.code) {
      case 'CONTEXT_OVERFLOW':
        return await this.compressContext(rejection);
      case 'RATE_LIMITED':
        return await this.exponentialBackoff(rejection);
      case 'UNAUTHENTICATED':
        return await this.refreshCredentials(rejection);
      default:
        throw new RecoveryFailure(
          `Unhandled MCP rejection: ${rejection.code}`,
          rejection
        );
    }
  }
}

The .env Leak That Cost Us $47,000

Environment variable management in AI systems is fundamentally different from traditional applications. Your agents are generating dynamic prompts, making external API calls, and sometimes producing output that contains sensitive information. A single misconfigured environment variable can leak API keys, database credentials, or customer PII through the agent's output stream.

I once deployed an agent that was supposed to analyze customer support tickets and summarize issues. Due to a typo in our Docker Compose configuration, the agent had access to the production database's connection string as an environment variable. The LLM, operating under instructions to "provide complete diagnostic information," included the full connection string in its response. This wasn't a vulnerability in the model—it was a failure in the separation between runtime configuration and model context.

The production patterns that prevented this in later systems:

1. Principle of Least Environment Only expose to the agent process the exact environment variables it needs for its current task. Use scoped environments per agent deployment rather than shared containers.

# Bad: Shared environment across all services
environment:
  - DATABASE_URL=${PROD_DB_URL}
  - API_KEY=${MASTER_API_KEY}
  - REDIS_URL=${REDIS_URL}

# Good: Scoped to agent task needs only
environment:
  - DB_CONNECTION=${AGENT_READONLY_CONN}
  - CACHE_ENDPOINT=${REDIS_URL}

2. Dynamic Credential Injection Rather than loading credentials from environment variables at startup, agents should fetch them dynamically through secure endpoints that validate the agent's current role and permissions.

3. Output Sanitization Layer Every agent output must pass through a sanitization filter that detects and redacts patterns matching common credential formats before reaching the user or downstream systems.

The 70-Line Loop Problem

The most insidious bug in production AI agents isn't a memory leak or a race condition—it's the infinite reasoning loop. When an agent enters a state where it repeatedly calls tools without making progress, the system consumes resources linearly while producing no value. A 70-line execution trace that never terminates is far more dangerous than an immediate crash.

I observed this pattern in agents that used overly permissive tool schemas. When an agent can call any tool, with any parameters, at any time, the state space becomes enormous. The LLM can get trapped in cycles like:

1. Call get_user_data → Returns empty result
2. Call validate_user_id → Returns false
3. Call refresh_user_session → Token still invalid
4. Call get_user_data again → Back to step 1

The solution requires explicit termination conditions at multiple layers:

Maximum Iteration Bounds Every agent execution should have a hard limit on tool invocations per turn. This isn't just a safeguard against runaway costs—it's a functional requirement for predictable latency.

Progress Detection Before allowing another tool call, verify that the agent's action has changed the system state meaningfully. If the last three invocations returned identical or cyclically redundant results, force a termination with a summary of the failed path.

Deduplication of Tool States Maintain a hash of recent tool call signatures. If the same tool is called with identical parameters within a sliding window, reject the duplicate and require the agent to branch into a different reasoning path.

class LoopDetector:
    def __init__(self, window_size=10):
        self.call_history = deque(maxlen=window_size)
        self.seen_signatures = set()
    
    def detect_loop(self, tool_call: ToolCall) -> bool:
        signature = self._hash_signature(tool_call)
        if signature in self.seen_signatures:
            return True
        self.seen_signatures.add(signature)
        return False
    
    def _hash_signature(self, call: ToolCall) -> str:
        return hashlib.sha256(
            f"{call.tool}:{json.dumps(call.params, sort_keys=True)}".encode()
        ).hexdigest()

Architectural Patterns That Survived Production

After deploying dozens of agent systems, several architectural patterns consistently outperformed others. These aren't theoretical—they're battle-tested against real MCP failures, credential leaks, and infinite loops.

The Circuit Breaker Pattern for Tool Calls Instead of treating tool failures as transient, implement circuit breakers that open after consecutive failures. This prevents the agent from exhausting API quotas or triggering rate limits on dependent services.

State Machine over Control Flow Represent agent logic as an explicit state machine rather than imperative control flow. Each state should have defined transitions, timeout behavior, and error handling. This makes debugging 70-line loops trivial—you can see exactly which state transition is repeating.

Observability as a First-Class Concern Every agent execution should emit structured telemetry: tool call sequences, latency distributions, rejection reasons, and state transitions. Without this, you're debugging blind when something goes wrong in production.

When to Stop Optimizing and Ship

The biggest mistake I see engineering teams make with AI agents is over-engineering the architecture before proving the core loop works. I've seen teams spend months building sophisticated MCP server clusters, circular dependency injection frameworks, and custom loop detection algorithms—only to discover the agent couldn't reliably perform its primary task.

The rule of thumb: if your agent can't complete its core task reliably in a controlled environment, production patterns won't save it. Start with the simplest possible architecture that handles the happy path, then add resilience patterns one at a time based on actual failure modes you observe in testing.

The Human Element in Production Agents

Finally, the lesson that took me longest to internalize: production AI agents are not just technical systems—they're socio-technical systems. Your agents interact with humans who have expectations, frustrations, and workarounds. An agent that technically works but confuses or frustrates users will fail in production faster than any technical debt.

The best production agents I've shipped share one characteristic: they make their limitations obvious. When an MCP call fails, the agent doesn't silently retry with slightly modified parameters—it tells the user what went wrong and suggests next steps. When it encounters a credential issue, it doesn't proceed with degraded access—it asks for re-authentication. Transparency builds trust; hidden failures erode it.

Production AI agent engineering is less about perfecting the LLM integration and more about building robust, observable, human-aware systems around it. The MCP rejections, .env leaks, and 70-line loops aren't edge cases—they're the curriculum. Master them, and your agents survive. Ignore them, and they'll teach you in the worst possible way: at 3 AM on a Friday.

For more practical guidance on building reliable AI systems, check out Tamiz's Insights, where deep technical analysis meets production reality.