
The 2026 Developer's Guide to Agent Fleets: Taming Unbounded Concurrency with Bounded Worker Pools
Build production-grade agent fleets with bounded worker pools, backpressure, circuit breakers, and cost-aware scheduling for concurrent AI workloads.
Agent fleets fail in a peculiar way: they work beautifully in a demo, then collapse under production traffic. The code looks reasonable. You spawn a task for every incoming request, every document chunk, every sub-agent. Each task calls a model, retries on failure, waits on a tool, and spawns more work. Throughput rises until something saturates: connection pools, rate limits, memory, your monthly budget, or the provider's willingness to serve you.
The fix is not to make agents faster. It is to make concurrency a bounded resource, not an emergent side effect of application logic. A bounded worker pool is the core primitive for 2026-era agent infrastructure: a fixed or elastic capacity budget that admits work only when capacity exists, applies backpressure when it does not, and sheds load deliberately instead of melting downstream.
This guide goes beneath the API surface. We will model agent execution as a queue-and-worker system, separate fan-out from fan-in, implement adaptive concurrency control, and add the observability and failure semantics required to operate agent fleets in production.
Table of Contents
- 1. The Concurrency Explosion
- 2. Core Architecture: Queues, Workers, and Adapters
- 3. Building a Bounded Worker Pool
- 4. Backpressure: The Most Important Missing Piece
- 5. Adaptive Concurrency: Finding the Right Limit
- 6. Circuit Breakers, Retries, and Dead Letter Queues
- 7. Fan-Out, Fan-In, and the Multi-Agent Shape
- 8. Observability: You Cannot Tune What You Cannot See
- 9. Cost and Budget Control
- 10. Production Deployment Patterns
- 11. Testing Agent Fleets
- 12. Anti-Patterns and Their Replacements
- 13. Architecture Decision Guide
- Frequently Asked Questions
1. The Concurrency Explosion
Consider a seemingly innocent document analysis pipeline:
async def process_document(doc):
chunks = await chunk_document(doc)
results = await asyncio.gather(*[
analyze_chunk(chunk) for chunk in chunks
])
return await synthesize(results)
One document might contain 500 chunks. Ten concurrent uploads produce 5,000 simultaneous model calls. The asyncio.gather call does not create parallelism; it creates 5,000 pending coroutines that will all attempt to execute as fast as the event loop and the HTTP client allow.
This pattern has three distinct problems.
First, downstream limits are not modeled. Model providers enforce token-per-minute (TPM), request-per-minute (RPM), and concurrency ceilings. If you exceed them, you receive 429 responses, latency spikes, or account-level throttling. Your application has no way to represent that limit, so it discovers it through failure.
Second, retries multiply load. A retry policy of three attempts across 5,000 tasks means up to 15,000 requests. Retries are correct in isolation, but when applied to an already-overloaded system, they create a retry storm that prolongs the outage.
Third, memory and tail latency degrade non-linearly. Each pending task holds its prompt, parsed response, and intermediate state. A million queued items is a million objects in memory, and the last item to finish waits behind the entire queue.
The architectural mistake is treating concurrency as a consequence of workload size. Concurrency should be a resource you allocate, like database connections or worker threads.
The mental model: agent fleets as queueing systems
An agent fleet is a distributed queueing system where the "server" is an LLM API with unknown, variable capacity. Classical queueing theory gives us the vocabulary:
| Concept | Agent fleet equivalent |
|---|---|
| Arrival rate (λ) | Requests submitted per second |
| Service rate (μ) | Requests completed per second |
| Queue | Pending task buffer, possibly durable |
| Server capacity | Concurrent model calls, tool workers |
| Blocking | Backpressure: producer waits for capacity |
| Rejection | Load shedding, 429 response, deferred job |
| Utilization (ρ) | In-flight work divided by permitted concurrency |
The critical insight: if arrival rate exceeds service rate, no amount of concurrency helps. Adding workers when the provider is already saturated just moves the queue. Your job is to keep utilization below 1 and to make the queue boundary explicit.
A worker pool is not a performance optimization. It is a contract about how many things may be in flight at once, and what happens when that contract cannot be met.
2. Core Architecture: Queues, Workers, and Adapters
A production agent fleet separates three concerns that demos usually conflate:
- The work source produces task descriptors. It knows what needs doing but not how much can run concurrently.
- The scheduler/pool owns the concurrency budget. It decides when work starts.
- The executor performs the actual agent run: model calls, tool invocations, retries.
This separation lets you swap an in-memory queue for Redis, SQS, or Kafka without touching agent logic. It also lets you apply different limits to different resource types. A fleet that calls an LLM and a fleet that shells out to a sandbox have entirely different capacity characteristics.
Defining the task contract
Start with a typed, serializable task descriptor. Do not pass live coroutines or open connections through the queue.
from dataclasses import dataclass, field
from typing import Any, Literal
TaskKind = Literal["llm", "tool", "embedding", "pipeline"]
@dataclass(frozen=True, slots=True)
class AgentTask:
task_id: str
kind: TaskKind
payload: dict[str, Any]
priority: int = 0
max_attempts: int = 3
timeout_s: float = 60.0
cost_limit_usd: float | None = None
metadata: dict[str, str] = field(default_factory=dict)
@dataclass(frozen=True, slots=True)
class AgentResult:
task_id: str
ok: bool
output: Any = None
error: str | None = None
attempts: int = 1
tokens_in: int = 0
tokens_out: int = 0
cost_usd: float = 0.0
duration_ms: int = 0
Two details matter here. max_attempts and timeout_s travel with the task so the executor's behavior is deterministic and replayable. The cost_limit_usd field lets the pool reject work that cannot possibly succeed within budget rather than discovering the overrun afterward.
Separating the pool from the executor
The pool should know nothing about LLMs. It accepts callables, so you can test it with a trivial function and use it for any kind of concurrent work.
import asyncio
import time
from collections.abc import Awaitable, Callable
from typing import Any, TypeVar
T = TypeVar("T")
Executor = Callable[[AgentTask], Awaitable[AgentResult]]
class BoundedWorkerPool:
"""A pool with a hard cap on concurrent executions.
The semaphore is the source of truth for concurrency. The queue is
unbounded by default, so producers never block; use `submit` with
`await` for backpressure, or rely on `try_submit` for load shedding.
"""
def __init__(
self,
executor: Executor,
*,
max_workers: int = 8,
max_queue: int | None = None,
name: str = "agent-pool",
) -> None:
if max_workers < 1:
raise ValueError("max_workers must be >= 1")
self._executor = executor
self._max_workers = max_workers
self._max_queue = max_queue
self._name = name
self._sem = asyncio.Semaphore(max_workers)
self._queue: asyncio.Queue[AgentTask | None] = asyncio.Queue()
self._workers: list[asyncio.Task[None]] = []
self._active = 0
self._completed = 0
self._failed = 0
async def start(self) -> None:
if self._workers:
return
self._workers = [
asyncio.create_task(self._worker_loop(i), name=f"{self._name}-{i}")
for i in range(self._max_workers)
]
async def stop(self, *, drain: bool = True, timeout: float = 30.0) -> None:
if drain:
await self._queue.join()
else:
while not self._queue.empty():
try:
self._queue.get_nowait()
self._queue.task_done()
except asyncio.QueueEmpty:
break
for _ in self._workers:
await self._queue.put(None)
await asyncio.wait_for(
asyncio.gather(*self._workers, return_exceptions=True),
timeout=timeout,
)
self._workers.clear()
async def submit(self, task: AgentTask) -> None:
"""Enqueue a task. Blocks (backpressure) if the queue is full."""
if self._max_queue is not None:
await self._queue.put(task)
else:
await self._queue.put(task)
async def _worker_loop(self, worker_id: int) -> None:
while True:
task = await self._queue.get()
try:
if task is None:
return
self._active += 1
try:
result = await asyncio.wait_for(
self._executor(task),
timeout=task.timeout_s,
)
self._completed += 1
except asyncio.TimeoutError:
self._failed += 1
except Exception:
self._failed += 1
finally:
self._active -= 1
finally:
self._queue.task_done()
This implementation is deliberately simple, and it is correct for the common case. But note two things it does not yet do: it does not return results to callers, and it does not apply rate limits or circuit breakers. We will add those.
Why a semaphore alone is not enough
You may be tempted to write this instead:
sem = asyncio.Semaphore(20)
async def guarded(task):
async with sem:
return await run_agent(task)
This caps concurrency, which is better than unbounded fan-out. It is still not a worker pool, for three reasons.
- No bounded queue. Callers still create every task up front. If you submit 100,000 tasks, you have 100,000 tasks in memory, each holding a payload. The semaphore limits execution, not admission.
- No natural priority or ordering. Priority requires a priority queue or a scheduler, not a semaphore.
- No lifecycle. There is no clean start, drain, or shutdown path, and no place to attach pool-level metrics.
The semaphore pattern is fine for a bounded fan-out of a few hundred items. For a fleet, use an explicit queue and workers.
3. Building a Bounded Worker Pool
Let's build a pool that returns results, supports priority, and exposes honest metrics. This is the version you would actually deploy.
import asyncio
import heapq
import itertools
import time
from dataclasses import dataclass, field
from typing import Any
@dataclass(order=True)
class _QueueItem:
sort_key: tuple[int, int] = field(compare=True)
task: AgentTask = field(compare=False)
@dataclass
class PoolStats:
submitted: int = 0
completed: int = 0
failed: int = 0
rejected: int = 0
active: int = 0
queued: int = 0
total_wait_ms: int = 0
class PriorityWorkerPool:
def __init__(
self,
executor: Executor,
*,
max_workers: int = 8,
max_queue: int = 1000,
name: str = "agent-pool",
) -> None:
self._executor = executor
self._max_workers = max_workers
self._max_queue = max_queue
self._name = name
self._heap: list[_QueueItem] = []
self._counter = itertools.count()
self._condition = asyncio.Condition()
self._results: dict[str, asyncio.Future[AgentResult]] = {}
self._workers: list[asyncio.Task[None]] = []
self._stats = PoolStats()
self._closed = False
async def start(self) -> None:
if self._workers:
return
self._workers = [
asyncio.create_task(self._worker_loop(i), name=f"{self._name}-{i}")
for i in range(self._max_workers)
]
async def submit(self, task: AgentTask) -> asyncio.Future[AgentResult]:
"""Admit a task or raise QueueFull.
Raises `asyncio.QueueFull` when the backlog exceeds max_queue.
"""
if self._closed:
raise RuntimeError("pool is closed")
async with self._condition:
if len(self._heap) >= self._max_queue:
self._stats.rejected += 1
raise asyncio.QueueFull(
f"{self._name}: queue full ({len(self._heap)}/{self._max_queue})"
)
future: asyncio.Future[AgentResult] = asyncio.get_running_loop().create_future()
self._results[task.task_id] = future
# Higher priority value sorts first; FIFO within a priority band.
heapq.heappush(
self._heap,
_QueueItem(sort_key=(-task.priority, next(self._counter)), task=task),
)
self._stats.submitted += 1
self._stats.queued = len(self._heap)
self._condition.notify()
return future
async def _worker_loop(self, worker_id: int) -> None:
while True:
async with self._condition:
while not self._heap and not self._closed:
await self._condition.wait()
if self._closed and not self._heap:
return
item = heapq.heappop(self._heap)
self._stats.queued = len(self._heap)
task = item.task
started = time.perf_counter()
self._stats.active += 1
try:
result = await asyncio.wait_for(
self._executor(task), timeout=task.timeout_s
)
self._stats.completed += 1
except asyncio.TimeoutError:
result = AgentResult(
task_id=task.task_id,
ok=False,
error=f"timeout after {task.timeout_s}s",
)
self._stats.failed += 1
except Exception as exc:
result = AgentResult(
task_id=task.task_id,
ok=False,
error=f"{type(exc).__name__}: {exc}\",