ProjectsAboutSkillsArticlesToolsContact
Back to Insights
AI & Machine Learning•The Hidden Non-Determinism of AI Agents: Why Identical Prompts Yield Different Results Across Hardware, and How to Build Deterministic Agent Pipelines•deep dive•November 8, 2026•12 min read

The Hidden Non-Determinism of AI Agents: Why Identical Prompts Yield Different Results Across Hardware, and How to Build Deterministic Agent Pipelines

Explore the hardware-level sources of AI agent non-determinism and learn practical techniques for building reproducible, deterministic agent pipelines.

T
Tamiz UddinFull-Stack Engineer

AI agents are notoriously unpredictable. You run the same prompt twice, and you get different outputs. But what if the differences aren't just due to sampling temperature? What if the very hardware your agent runs on—CPU, GPU, memory—is silently corrupting reproducibility? This article dives into the hidden non-determinism of AI agents across hardware and presents a systematic approach to building deterministic agent pipelines.

Table of Contents

1. The Reproducibility Crisis in AI Agents

AI agents are complex systems that combine LLMs, retrieval, tool calling, and orchestration logic. In production, they must behave consistently across environments—development, staging, and multiple cloud regions. Yet, the same prompt can yield different answers on different machines, leading to flaky tests, inconsistent user experiences, and debugging nightmares.

Most engineers blame sampling randomness. But even with temperature=0, many agents remain non-deterministic. The root cause often lies deeper: in the hardware and low-level software stack. This article uncovers those hidden sources and shows you how to tame them.

2. Sources of Non-Determinism: From Silicon to Software

2.1 GPU Floating-Point Non-Associativity

Floating-point arithmetic is not associative. (a + b) + c is not always equal to a + (b + c) due to rounding. GPUs perform massively parallel reductions (e.g., summing gradients or attention scores) where the order of operations is not fixed. This means that even with the same model weights and input, the output can differ slightly across runs on the same GPU, and more significantly across different GPU architectures (e.g., NVIDIA A100 vs. V100) that use different reduction tree structures.

2.2 CUDA / cuBLAS Non-Deterministic Kernels

NVIDIA's cuBLAS and cuDNN libraries include multiple kernel implementations for the same operation. By default, they use heuristics to select the fastest kernel for a given input size and hardware. This selection can change based on memory alignment, tensor layout, or even the current GPU clock. Some kernels are non-deterministic: they use atomic operations or parallel reductions that produce different rounding orders. The result is bit-level differences in logits, which can flip token choices when the probability distribution is near a threshold.

2.3 CPU Instruction Set Variations

Even on CPU-only inference, differences in instruction sets (e.g., AVX2, AVX-512, FMA) can cause different floating-point results. A model compiled with AVX-512 on one machine and AVX2 on another may produce slightly different outputs. Similarly, the order of operations in SIMD vectorization can change rounding.

2.4 Memory Allocation and Parallel Scheduling

The OS and runtime (Python, PyTorch) allocate memory in ways that affect pointer alignment and cache behavior. For parallel operations, the thread scheduling order can influence the order of floating-point operations. For example, PyTorch's torch.nn.DataParallel or torch.compile may split tensors across threads differently based on system load, causing non-determinism.

2.5 Software-Level Randomness

Beyond hardware, there are software sources: random seeds for dropout, data shuffling, and sampling. But there are also less obvious ones: hash randomization (PYTHONHASHSEED), non-deterministic sorting algorithms (e.g., Timsort is stable but not deterministic across runs if the input order varies), and the use of set or dict iteration order in Python, which depends on hash seeds.

3. How Hardware Differences Manifest in Agent Behavior

In an agent pipeline, these low-level differences accumulate. An LLM call might produce logits that differ by 1e-6. At temperature=0, the argmax is usually the same, but not always. If two tokens have nearly equal probabilities, a tiny difference can flip the choice. That flip changes the agent's next action: it might call a different tool, retrieve different documents, or produce a different final answer.

Moreover, agents often use retrieval systems (e.g., vector databases) that rely on similarity search. Those searches are also subject to floating-point non-determinism, especially when using approximate nearest neighbor (ANN) algorithms. A slight difference in embeddings (due to GPU non-determinism) can change the top-k results, cascading into completely different agent trajectories.

4. Building a Deterministic Agent Pipeline: A Practical Guide

Here's a systematic approach to achieve reproducibility in agent pipelines.

4.1 Pin the Environment

Start by pinning every dependency version and the hardware configuration. Use Docker to containerize the runtime, and pin the base image, CUDA version, cuDNN version, and Python package versions. For hardware, if you need cross-machine determinism, you must standardize on a specific GPU model and driver version.

dockerfile
# Dockerfile
FROM nvidia/cuda:12.1.0-cudnn8-runtime-ubuntu22.04
RUN pip install --no-cache-dir \
    torch==2.1.0 \
    transformers==4.35.0 \
    langchain==0.1.0

4.2 Control the Randomness

Set all random seeds, but also set PYTHONHASHSEED to a fixed value to make Python hash-based iteration deterministic.

python
import os
import random
import numpy as np
import torch

SEED = 42
os.environ['PYTHONHASHSEED'] = str(SEED)
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)

4.3 Force Deterministic Execution

PyTorch provides flags to force deterministic algorithms. Enable them, but be aware of potential performance costs.

python
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)

For TensorFlow, set the following:

python
import tensorflow as tf
tf.keras.utils.set_random_seed(42)
tf.config.experimental.enable_op_determinism()

Note that not all operations have deterministic implementations. If you use a custom op that is non-deterministic, the code may raise an error. You can also set CUBLAS_WORKSPACE_CONFIG=:4096:8 to make cuBLAS deterministic.

4.4 Cache and Version Everything

In agent pipelines, caching is a powerful tool. Cache the outputs of deterministic steps (like embedding generation) so that if the input is identical, the output is reused. Use a content-addressed cache (e.g., a key-value store keyed by hash of input and model version). Also version your models and prompts. A change in the model or prompt should invalidate the cache.

python
import hashlib
import json
import redis

cache = redis.Redis()

def get_cached_or_compute(key_prefix, input_data, compute_fn):
    key = key_prefix + hashlib.sha256(json.dumps(input_data, sort_keys=True).encode()).hexdigest()
    cached = cache.get(key)
    if cached:
        return json.loads(cached)
    result = compute_fn(input_data)
    cache.set(key, json.dumps(result))
    return result

4.5 Implement a Deterministic Fallback Layer

For critical components like the LLM call itself, you can implement a fallback that uses a fully deterministic inference engine. For example, you can use llama.cpp with integer quantization and deterministic sampling (greedy) on CPU. This may be slower but guarantees bit-level reproducibility across machines with the same CPU architecture.

Alternatively, you can run the LLM on a fixed GPU and disable multi-threading in the inference engine. For HuggingFace, you can set torch.set_num_threads(1) to reduce thread-scheduling non-determinism.

5. Case Study: A Deterministic RAG Agent

Let's build a retrieval-augmented generation (RAG) agent with deterministic behavior.

Architecture:

  1. Embed the query using a sentence transformer.
  2. Retrieve top-k documents from a vector DB.
  3. Construct a prompt with the documents and the original query.
  4. Generate an answer with an LLM.

Deterministic implementation:

python
from sentence_transformers import SentenceTransformer
from langchain.vectorstores import FAISS
from langchain.llms import HuggingFacePipeline
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch
import os

# Set determinism
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

# Load models with fixed seeds
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
# For FAISS, use exact search (index_flat) instead of IVF to avoid ANN non-determinism
vectorstore = FAISS.load_local('index', embedding_model, allow_dangerous_deserialization=True)
vectorstore.index = vectorstore.index.reconstruct_n()  # force exact search? Actually use index_flat

# Use deterministic LLM pipeline (greedy decoding)
tokenizer = AutoTokenizer.from_pretrained('gpt2')
model = AutoModelForCausalLM.from_pretrained('gpt2')
pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, max_new_tokens=50, do_sample=False)
llm = HuggingFacePipeline(pipeline=pipe)

query = "What is the capital of France?"
docs = vectorstore.similarity_search(query, k=3)
context = "\n".join([d.page_content for d in docs])
prompt = f"Context:\n{context}\n\nQuestion: {query}\nAnswer:"
print(llm(prompt))

In this example, we use exact FAISS search (flat index) to avoid ANN non-determinism. The LLM uses greedy decoding (do_sample=False) to avoid sampling randomness. The PyTorch deterministic flags ensure that the model's forward pass is deterministic on a given GPU.

To make it reproducible across GPUs, you would need to run on the same GPU model or use a CPU-only inference with quantized models. For cross-hardware determinism, consider using llama.cpp with a fixed thread count and CPU instruction set.

6. Trade-offs and When Determinism Isn't Worth It

Forcing determinism often comes at a cost:

  • Performance: Deterministic algorithms may be slower (e.g., disabling cuDNN autotuning, using single-threaded inference).
  • Portability: You may be locked to a specific hardware configuration.
  • Flexibility: Some non-deterministic operations (like dropout during training) are necessary.

Determinism is most critical for:

  • Testing and CI/CD: You need stable test results.
  • Auditing and compliance: You need to reproduce decisions.
  • Debugging: You need to trace issues.

In contrast, for user-facing creative tasks, a little randomness can be desirable. In those cases, focus on controlling the seed and documenting the expected variance.

7. Conclusion

AI agent non-determinism is not just a software issue—it's a hardware issue. Floating-point arithmetic, parallel kernel selection, and memory scheduling all contribute to subtle variations that can cascade into completely different agent behaviors. By understanding these sources and implementing a deterministic pipeline—pinning the environment, setting seeds, forcing deterministic algorithms, caching, and using deterministic fallbacks—you can achieve reproducible agents. While there are trade-offs, the benefits in testing and production reliability are substantial.

Frequently Asked Questions

Q: Does setting temperature=0 guarantee deterministic LLM output?

A: No. Even with greedy decoding, the underlying floating-point computations can vary across hardware and even across runs on the same hardware due to non-deterministic kernels. You need to also set deterministic flags and pin the environment.

Q: Can I get bit-level identical results across different GPU models?

A: Rarely. Different GPU architectures use different floating-point units and reduction ordersING. For bit-level reproducibility, you may need to use CPU inference with a fixed instruction set, or use integer quantization.

Q: What is the biggest source of non-determinism in agent pipelines?

A: Often it's the LLM inference itself due to GPU kernel non-determinismastra. However, retrieval steps (e.g., vector search) and tool-call selections can also be significant. Address all layers for full determinism.

For more insights on AI engineering, visit Tamiz's Insights.

Building a Deterministic Agent Pipeline: A Practical Guide

Now that we understand the sources of non-determinism, let's construct a reference architecture that mitigates them. We'll build a Python-based agent that combines a language model, a retrieval tool, and a structured output validator—all pinned to reproducible settings.

Step 1: Environment Pinning

First, lock down the software and hardware variables. Use requirements.txt with exact versions, and set environment variables before importing any ML libraries.

python
# config.py
import os
import random
import numpy as np
import torch

def set_seed(seed: int = 42):
    """Set all possible random seeds for reproducibility."""
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    # Disable cuDNN autotuning (which can introduce non-determinism)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    # For HuggingFace transformers
    os.environ["TRANSFORMERS_DETERMINISTIC"] = "1"
    os.environ["TOKENIZERS_PARALLELISM"] = "false"
    # Force deterministic algorithms in PyTorch
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

# Call this before any model loading
set_seed(42)

Note on hardware: Even with these seeds, GPU kernels can produce different floating-point results across architectures (e.g., A100 vs. V100). For true bit-level determinism, you must run on the same GPU model and driver version. If you need cross-hardware reproducibility, consider using a fixed-precision inference engine like ONNX Runtime with int8 quantization—but this sacrifices quality.

Step 2: Model Loading with Fixed Configurations

Load your LLM with all generation parameters explicitly set. Never rely on defaults.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

def load_deterministic_model(model_name: str = "microsoft/phi-2"):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float32,  # float32 for cross-GPU stability
        low_cpu_mem_usage=True,
    )
    model.eval()
    return model, tokenizer

def generate_deterministic(model, tokenizer, prompt: str, max_new_tokens: int = 100):
    inputs = tokenizer(prompt, return_tensors="pt")
    # All generation parameters are explicitly set
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=False,          # Greedy decoding—no randomness
        temperature=1.0,          # Irrelevant when do_sample=False
        top_p=1.0,
        top_k=50,
        repetition_penalty=1.0,
        num_beams=1,              # Greedy, not beam search
        use_cache=True,           # Deterministic KV-cache
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Critical insight: Setting do_sample=False (greedy decoding) eliminates the largest source of token-level randomness. If you need sampling (e.g., for creative tasks), you must set temperature=0—this effectively becomes greedy but is mathematically identical across hardware.

Step 3: Deterministic Retrieval

Vector search is inherently non-deterministic due to approximate nearest neighbor (ANN) indexes. For reproducibility, use exact search or a fixed seed.

python
import faiss
import numpy as np

class DeterministicRetriever:
    def __init__(self, embeddings: np.ndarray):
        self.dim = embeddings.shape[1]
        # Use exact search (Flat) instead of IVF/HNSW
        self.index = faiss.IndexFlatIP(self.dim)  # Inner product
        self.index.add(embeddings)
        
    def retrieve(self, query_embedding: np.ndarray, k: int = 3):
        # Exact search is deterministic—no approximation
        scores, indices = self.index.search(query_embedding, k)
        return indices[0].tolist(), scores[0].tolist()

If you must use an ANN index (e.g., for scale), set the random seed and the number of probes:

python
# For HNSW, seed the index construction
index = faiss.IndexHNSWFlat(dim, 32)
index.hnsw.efConstruction = 200
index.hnsw.efSearch = 64
# faiss uses a random number generator; seed it
faiss.omp_set_num_threads(1)  # Single-thread for determinism

Step 4: Structured Output with Validation

To ensure the agent's final output is not only deterministic but also schema-compliant, use a validator that retries with a fixed strategy.

python
from pydantic import BaseModel, ValidationError
import json

class AgentOutput(BaseModel):
    answer: str
    confidence: float
    sources: list[str]

def validate_and_repair(output_text: str, max_retries: int = 2):
    """Attempt to parse, and if failed, use a deterministic repair prompt."""
    for attempt in range(max_retries):
        try:
            data = json.loads(output_text)
            return AgentOutput(**data)
        except (json.JSONDecodeError, ValidationError):
            # Deterministic repair: ask the model to fix JSON only
            repair_prompt = f"Fix this JSON to match schema: {AgentOutput.schema_json()}\nOriginal: {output_text}\nReturn only fixed JSON."
            output_text = generate_deterministic(model, tokenizer, repair_prompt, max_new_tokens=200)
    raise RuntimeError("Failed to produce valid output after retries")

Step 5: Full Agent Pipeline with Deterministic Orchestration

Now assemble everything into a single deterministic pipeline.

python
class DeterministicAgent:
    def __init__(self, retriever: DeterministicRetriever, model, tokenizer):
        self.retriever = retriever
        self.model = model
        self.tokenizer = tokenizer
        
    def run(self, query: str, max_steps: int = 2) -> AgentOutput:
        # Step 1: Embed query (use a deterministic embedding model)
        query_embedding = self._embed(query)
        
        # Step 2: Retrieve (exact search)
        indices, scores = self.retriever.retrieve(query_embedding)
        contexts = [self._get_doc(i) for i in indices]
        
        # Step 3: Build prompt with fixed template
        prompt = self._build_prompt(query, contexts)
        
        # Step 4: Generate (greedy)
        raw_output = generate_deterministic(
            self.model, self.tokenizer, prompt, max_new_tokens=200
        )
        
        # Step 5: Validate
        return validate_and_repair(raw_output)
    
    def _embed(self, text: str) -> np.ndarray:
        # Use a deterministic embedding function (e.g., sentence-transformers with fixed seed)
        # For brevity, assume this returns a fixed vector given the same text
        pass

Testing for Determinism

Write a test that runs the pipeline multiple times and asserts identical outputs.

python
def test_determinism(agent: DeterministicAgent, query: str, runs: int = 5):
    outputs = [agent.run(query) for _ in range(runs)]
    first = outputs[0]
    for out in outputs[1:]:
        assert out == first, f"Non-deterministic: {out} != {first}"
    print("✓ Pipeline is deterministic across runs")

# Run on CPU and GPU to compare
# Note: Cross-hardware determinism requires identical GPU models

Advanced: Handling Tool-Call Selection

If your agent uses tools (e.g., function calling), the model's choice of which tool to call can vary. To make this deterministic:

  1. Use greedy decoding for the tool-selection step.
  2. Define a strict tool schema with no ambiguous descriptions.
  3. Cache tool results based on input hash—if the same tool is called with the same arguments, return the cached result.
python
tool_cache = {}

def call_tool_deterministic(tool_name: str, args: dict):
    import hashlib
    key = hashlib.sha256(f"{tool_name}:{json.dumps(args, sort_keys=True)}".encode()).hexdigest()
    if key in tool_cache:
        return tool_cache[key]
    result = execute_tool(tool_name, args)
    tool_cache[key] = result
    return result

Real-World Trade-offs and Caveats

  • GPU vs. CPU: Even with all seeds set, GPU results may differ from CPU due to different floating-point accumulation orders. For cross-hardware determinism, run on the same architecture or use CPU (slower but more consistent).
  • Library updates: A patch version of PyTorch or Transformers can change kernel implementations. Pin all versions in your requirements.txt and use a lock file.
  • Cloud environment: If you use managed inference APIs (e.g., OpenAI), you have no control over the backend. For deterministic agents, self-host your models.
  • Temperature > 0: If your use case requires sampling, you cannot achieve bit-level determinism. Instead, use a fixed random seed and accept that different hardware will produce different results—but the same hardware will produce the same results.

Conclusion: Determinism is a Design Choice, Not a Default

The hidden non-determinism in AI agents stems from a stack of interdependent layers: hardware floating-point behavior, library kernels, random seeds, decoding strategies, and retrieval approximations. By systematically addressing each layer—pinning environments, using greedy decoding, exact search, and validated outputs—you can build agents that produce identical results across runs on the same hardware.

For cross-hardware reproducibility, the gold standard is running on identical GPU models with identical driver and library versions. If that's impossible, accept that you can achieve statistical determinism (same distribution of outputs) but not bit-level determinism.

The effort is worth it for production systems where auditability, testing, and user trust depend on consistent behavior. Start by instrumenting your current pipeline to detect non-determinism, then apply the fixes incrementally. Your users—and your regression tests—will thank you.

For more insights on AI engineering, visit Tamiz's Insights.