"My Agent Refused 96 Times": Building Self-Editing Agents with Hard Failure Modes

In the early days of shipping LLM-based agents, we optimized for output volume. If the model could not find the answer, it often generated a plausible one anyway. This is the "yes-man" problem. In critical systems—financial auditing, code generation, or compliance checks—this creates a dangerous class of errors: confident hallucinations.

Recently, a senior ML engineer shared a benchmark result from a production support agent: the system refused to answer a valid question 96 times in a test set. While the product team initially flagged this as a failure rate, the engineering review revealed the opposite. In 96% of those cases, the model correctly identified that the retrieved context was insufficient or contradictory. It chose the hard failure mode: refusal.

This is a pivotal shift in how we design autonomous systems. We are moving from probabilistic output to deterministic verification. This article explores how to architect agents with hard failure modes, treating "I don't know" as a first-class return type rather than an exception to be suppressed.

The Cost of Politeness in LLM Agents

Large Language Models are trained on corpora where the goal is often to be helpful and coherent. Consequently, they exhibit a strong bias toward generating a response, even when the semantic signal is absent. This is known as sycophancy in evaluation contexts.

When you build a RAG (Retrieval-Augmented Generation) agent, the default pipeline is:

1. Retrieve: Fetch context chunks from a vector store.
2. Synthesize: Inject context into the prompt.
3. Generate: Ask the model to answer.

The flaw lies in step 3. Without a gatekeeper, if the retrieved chunks contain irrelevant data (due to embedding similarity thresholds being too loose), the model will attempt to bridge the gap with internal parametric knowledge, leading to hallucination.

Consider the following naive implementation, which represents the "volume-first" architecture:

# DANGEROUS: No verification gate
async def get_answer(question: str) -> str:
    context = await retrieve_context(question)
    prompt = f"Context:\n{context}\n\nQuestion: {question}"
    return await llm.complete(prompt)

In a high-stakes environment, this function might return "The SQL query should use JOIN on table B" even if table B was never mentioned in the context. The agent has failed, but the API client sees a successful 200 OK with text. The error is silent.

Defining Hard Failure Modes

A hard failure mode is a deterministic exit path where the agent explicitly signals that it cannot fulfill the request based on verifiable criteria, rather than probabilistic guessing.

This concept borrows heavily from systems programming (e.g., Rust’s Result<T, E> or Go’s error handling). The agent must validate its own output before returning it to the user. If validation fails, the agent does not return a "best effort" string; it returns a structured Failure object.

Why Self-Editing?

The term "self-editing" implies a secondary LLM call or a deterministic check that reviews the primary output. It is a form of self-correction or refusal.

There are two main types of self-editing gates:

1. Structural Gates: Checks for schema validity, length, or presence of required fields (deterministic).
2. Semantic Gates: A second LLM (or the same one) acts as a critic, verifying if the answer is supported by the context (probabilistic but grounded).

Architecture: The Verification Loop

To build an agent that "refuses 96 times," we need a multi-step pipeline where the output of the generator is the input to a verifier. This is often called a critic-loop or verification-based generation.

Step 1: Deterministic Schema Validation

Before the LLM even generates text, define the strict shape of the expected answer. If your agent extracts data, it must output JSON. Use a library like Pydantic or Zod to enforce this.

If the LLM outputs malformed JSON, the parser fails. This is not a hallucination; this is a syntax error. The agent should immediately fall back to a refusal state or retry with a stricter prompt, rather than attempting to parse a broken string.

Step 2: Grounding Verification (The "Refusal" Logic)

This is the core mechanism. We must introduce a verification step that checks if the generated answer is entailed by the context.

We can implement this using a Logit Bias approach or a Second-Pass LLM. The second-pass approach is more robust for complex reasoning.

The Second-Pass Verifier Pattern

Instead of trusting the first completion, we ask the LLM to evaluate its own work. The prompt structure changes from:

Q: [Question] A: [Answer]

To:

Context: [Context] Q: [Question] A: [Draft Answer] Verify if A is fully supported by Context. Output TRUE or FALSE.

If the verifier outputs FALSE, the agent enters the failure mode. It does not return the draft. It returns a standard refusal message.

Here is a Python implementation of a self-editing agent using a verification loop:

from pydantic import BaseModel
import openai
import json

class VerificationResult(BaseModel):
    is_supported: bool
    reasoning: str

class AgentResponse(BaseModel):
    answer: str | None
    status: str # "success" or "refused"
    reason: str | None

def verify_answer(context: str, question: str, draft: str) -> VerificationResult:
    prompt = f"""
    You are a rigorous fact-checker. 
    
    Context:
    {context}
    
    Question: {question}
    Draft Answer: {draft}
    
    Task: Determine if the Draft Answer is strictly supported by the Context.
    - If the answer is in the context, output TRUE.
    - If the answer is a hallucination or not mentioned, output FALSE.
    - Do not use outside knowledge.
    
    Return JSON: {{"is_supported": true/false, "reasoning": "..."}}
    """
    response = openai.chat.completions.create(
        model="gpt-4o-mini", # Cost-effective verifier
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"}
    )
    return VerificationResult(**json.loads(response.choices[0].message.content))

async def self_editing_agent(question: str, context: str) -> AgentResponse:
    # 1. Generate draft
    draft_prompt = f"Context:\n{context}\n\nAnswer: {question}"
    draft_resp = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": draft_prompt}]
    )
    draft_answer = draft_resp.choices[0].message.content
    
    # 2. Verify
    verdict = verify_answer(context, question, draft_answer)
    
    # 3. Decide
    if verdict.is_supported:
        return AgentResponse(answer=draft_answer, status="success", reason=None)
    else:
        # HARD FAILURE MODE: Refuse
        return AgentResponse(
            answer=None, 
            status="refused", 
            reason=f"Insufficient context to answer confidently: {verdict.reason}"
        )

Step 3: Handling the Refusal

When the agent refuses, the response must be explicit. In the example above, the status is refused. The frontend or downstream service should treat this differently from a success response.

If you have 96 refusals in a test suite, analyze them. Are they true positives (the question was indeed unanswerable)? If so, your system is working correctly. If they are false positives (the answer was in the context but the verifier missed it), you need to tune the verifier prompt or lower the temperature of the generator.

Implementing Retry with Degradation

A hard failure mode does not always mean immediate termination. In many systems, a "refusal" triggers a fallback strategy. This is where self-editing becomes powerful.

If the verifier rejects the answer, the agent can attempt to re-generate with modified constraints. For example:

1. First Attempt: Full creative liberty.
2. Verification Fails: Re-prompt with "Based strictly on the provided context, do not hallucinate. If you cannot find the answer, say 'Not found'."
3. Second Verification Fails: Final refusal.

This adds latency but drastically improves precision. For every 100 requests, you might accept 85, retry and accept 10, and refuse 5. The 96 refusals mentioned in the industry anecdotes often come from systems that skip the retry and go straight to refusal, which is actually safer.

Metrics That Matter: Precision vs. Recall

Traditional accuracy metrics are misleading here. You want to optimize for Precision (when the agent speaks, it is right) over Recall (the agent answers everything).

The Trade-off Curve

• Low Refusal Rate: High recall, low precision. The agent guesses often. Hallucinations rise.
• High Refusal Rate: Low recall, high precision. The agent is stingy but accurate. Users may perceive it as "dumb" because it says "I don't know" often.

The "96 refusals" story is compelling because it highlights that the cost of a false positive in an AI agent is often orders of magnitude higher than the cost of a false negative (refusal). In a medical diagnosis bot, a false positive (hallucinating a treatment) is catastrophic. A false negative (refusing to answer) is merely inconvenient.

Production Patterns for Self-Correction

Beyond the basic verifier loop, there are advanced patterns used in production-grade agents (like those at companies building autonomous coding assistants).

1. Tool-Use Verification

If your agent uses tools (APIs, databases), the result of the tool call is the ground truth. You can verify that the LLM's summary of the tool result matches the actual tool output.

For example, if the agent calls get_balance(user_id) and receives 0.00, the LLM must not write "The user has $100 in their account." A simple string comparison or checksum can verify this. This is a deterministic hard failure mode.

2. Chain-of-Thought Pruning

Encourage the model to output its reasoning steps (CoT) before the final answer. The verifier can check the reasoning chain for logical gaps. If the reasoning is flawed but the answer is correct (lucky guess), the verifier should still reject it. This enforces explainable accuracy.

3. Temperature Cycling

During the self-editing loop, you can dynamically adjust the temperature. If the first generation is rejected, retry with temperature=0 to minimize variance and force the model to stick closer to the context distribution.

Common Pitfalls

The "Verifer Paradox"

If your verifier is just another LLM call, it can also hallucinate. It might falsely reject a correct answer. To mitigate this:

• Use a smaller, faster, and cheaper model for verification (e.g., gpt-4o-mini or claude-3-haiku).
• Use multiple verifiers and vote (ensemble).
• Combine LLM verification with deterministic keyword matching for critical entities.

Latency Costs

Self-editing adds at least one extra LLM round-trip. This doubles latency and cost.

• Mitigation: Cache frequent questions.
• Mitigation: Only run the verifier on "low confidence" scores (if your base model provides probability scores).
• Mitigation: Use the verifier only for high-stakes queries, not low-stakes chitchat.

Conclusion

The story of the agent refusing 96 times is a triumph of engineering integrity over product vanity. In the short term, a high refusal rate looks bad on a dashboard. In the long term, it builds trust. Users learn that when the agent speaks, they can trust it implicitly.

Building self-editing agents with hard failure modes requires a shift in mindset: from generating content to verifying correctness. By implementing structured verification loops, enforcing schema strictness, and treating refusal as a valid state, we can build AI systems that are not just intelligent, but reliable.

As we push toward more autonomous agents, the ability to say "no" is the defining characteristic of robustness. The code samples and patterns provided here serve as a foundation for transitioning your agent from a probabilistic chatbot to a deterministic worker.

Frequently Asked Questions

Q: How do I measure if my refusal rate is too high? A: Establish a "Golden Set" of questions with known answerable/unanswerable labels. Calculate your Refusal Accuracy: (True Refusals / Total Refusals). If this is >95%, your refusal rate is healthy, regardless of the absolute number. If it is <80%, your verifier is too aggressive.

Q: Can I use this pattern with non-LLM components? A: Yes. Hard failure modes apply to any stochastic system. If you have a heuristic classifier that sometimes confuses two classes, you can add a confidence threshold. If confidence < T, refuse. This is the same logic, just deterministic instead of LLM-based.

Q: What is the best model for the verifier? A: The verifier does not need to be creative; it needs to be precise. Smaller models like gpt-4o-mini, claude-3-haiku, or even quantized local models like llama-3.2-3b often perform surprisingly well at verification tasks because they follow instructions closely without adding flair. Benchmark a few candidates on your specific data.