Back to Insights
AI & Machine Learning•The 2026 Developer's Guide to Agent Fleets: Taming Unbounded Concurrency with Bounded Worker Pools•deep dive•November 20, 2026•9 min read

The 2026 Developer's Guide to Agent Fleets: Taming Unbounded Concurrency with Bounded Worker Pools

Build production-grade agent fleets with bounded worker pools, backpressure, circuit breakers, and cost-aware scheduling for concurrent AI workloads.

T
Tamiz UddinFull-Stack Engineer

Agent fleets fail in a peculiar way: they work beautifully in a demo, then collapse under production traffic. The code looks reasonable. You spawn a task for every incoming request, every document chunk, every sub-agent. Each task calls a model, retries on failure, waits on a tool, and spawns more work. Throughput rises until something saturates: connection pools, rate limits, memory, your monthly budget, or the provider's willingness to serve you.

The fix is not to make agents faster. It is to make concurrency a bounded resource, not an emergent side effect of application logic. A bounded worker pool is the core primitive for 2026-era agent infrastructure: a fixed or elastic capacity budget that admits work only when capacity exists, applies backpressure when it does not, and sheds load deliberately instead of melting downstream.

This guide goes beneath the API surface. We will model agent execution as a queue-and-worker system, separate fan-out from fan-in, implement adaptive concurrency control, and add the observability and failure semantics required to operate agent fleets in production.

Table of Contents

1. The Concurrency Explosion

Consider a seemingly innocent document analysis pipeline:

python
async def process_document(doc):
    chunks = await chunk_document(doc)
    results = await asyncio.gather(*[
        analyze_chunk(chunk) for chunk in chunks
    ])
    return await synthesize(results)

One document might contain 500 chunks. Ten concurrent uploads produce 5,000 simultaneous model calls. The asyncio.gather call does not create parallelism; it creates 5,000 pending coroutines that will all attempt to execute as fast as the event loop and the HTTP client allow.

This pattern has three distinct problems.

First, downstream limits are not modeled. Model providers enforce token-per-minute (TPM), request-per-minute (RPM), and concurrency ceilings. If you exceed them, you receive 429 responses, latency spikes, or account-level throttling. Your application has no way to represent that limit, so it discovers it through failure.

Second, retries multiply load. A retry policy of three attempts across 5,000 tasks means up to 15,000 requests. Retries are correct in isolation, but when applied to an already-overloaded system, they create a retry storm that prolongs the outage.

Third, memory and tail latency degrade non-linearly. Each pending task holds its prompt, parsed response, and intermediate state. A million queued items is a million objects in memory, and the last item to finish waits behind the entire queue.

The architectural mistake is treating concurrency as a consequence of workload size. Concurrency should be a resource you allocate, like database connections or worker threads.

The mental model: agent fleets as queueing systems

An agent fleet is a distributed queueing system where the "server" is an LLM API with unknown, variable capacity. Classical queueing theory gives us the vocabulary:

ConceptAgent fleet equivalent
Arrival rate (λ)Requests submitted per second
Service rate (μ)Requests completed per second
QueuePending task buffer, possibly durable
Server capacityConcurrent model calls, tool workers
BlockingBackpressure: producer waits for capacity
RejectionLoad shedding, 429 response, deferred job
Utilization (ρ)In-flight work divided by permitted concurrency

The critical insight: if arrival rate exceeds service rate, no amount of concurrency helps. Adding workers when the provider is already saturated just moves the queue. Your job is to keep utilization below 1 and to make the queue boundary explicit.

A worker pool is not a performance optimization. It is a contract about how many things may be in flight at once, and what happens when that contract cannot be met.

2. Core Architecture: Queues, Workers, and Adapters

A production agent fleet separates three concerns that demos usually conflate:

  1. The work source produces task descriptors. It knows what needs doing but not how much can run concurrently.
  2. The scheduler/pool owns the concurrency budget. It decides when work starts.
  3. The executor performs the actual agent run: model calls, tool invocations, retries.

This separation lets you swap an in-memory queue for Redis, SQS, or Kafka without touching agent logic. It also lets you apply different limits to different resource types. A fleet that calls an LLM and a fleet that shells out to a sandbox have entirely different capacity characteristics.

Defining the task contract

Start with a typed, serializable task descriptor. Do not pass live coroutines or open connections through the queue.

python
from dataclasses import dataclass, field
from typing import Any, Literal

TaskKind = Literal["llm", "tool", "embedding", "pipeline"]

@dataclass(frozen=True, slots=True)
class AgentTask:
    task_id: str
    kind: TaskKind
    payload: dict[str, Any]
    priority: int = 0
    max_attempts: int = 3
    timeout_s: float = 60.0
    cost_limit_usd: float | None = None
    metadata: dict[str, str] = field(default_factory=dict)

@dataclass(frozen=True, slots=True)
class AgentResult:
    task_id: str
    ok: bool
    output: Any = None
    error: str | None = None
    attempts: int = 1
    tokens_in: int = 0
    tokens_out: int = 0
    cost_usd: float = 0.0
    duration_ms: int = 0

Two details matter here. max_attempts and timeout_s travel with the task so the executor's behavior is deterministic and replayable. The cost_limit_usd field lets the pool reject work that cannot possibly succeed within budget rather than discovering the overrun afterward.

Separating the pool from the executor

The pool should know nothing about LLMs. It accepts callables, so you can test it with a trivial function and use it for any kind of concurrent work.

python
import asyncio
import time
from collections.abc import Awaitable, Callable
from typing import Any, TypeVar

T = TypeVar("T")

Executor = Callable[[AgentTask], Awaitable[AgentResult]]


class BoundedWorkerPool:
    """A pool with a hard cap on concurrent executions.

    The semaphore is the source of truth for concurrency. The queue is
    unbounded by default, so producers never block; use `submit` with
    `await` for backpressure, or rely on `try_submit` for load shedding.
    """

    def __init__(
        self,
        executor: Executor,
        *,
        max_workers: int = 8,
        max_queue: int | None = None,
        name: str = "agent-pool",
    ) -> None:
        if max_workers < 1:
            raise ValueError("max_workers must be >= 1")
        self._executor = executor
        self._max_workers = max_workers
        self._max_queue = max_queue
        self._name = name
        self._sem = asyncio.Semaphore(max_workers)
        self._queue: asyncio.Queue[AgentTask | None] = asyncio.Queue()
        self._workers: list[asyncio.Task[None]] = []
        self._active = 0
        self._completed = 0
        self._failed = 0

    async def start(self) -> None:
        if self._workers:
            return
        self._workers = [
            asyncio.create_task(self._worker_loop(i), name=f"{self._name}-{i}")
            for i in range(self._max_workers)
        ]

    async def stop(self, *, drain: bool = True, timeout: float = 30.0) -> None:
        if drain:
            await self._queue.join()
        else:
            while not self._queue.empty():
                try:
                    self._queue.get_nowait()
                    self._queue.task_done()
                except asyncio.QueueEmpty:
                    break
        for _ in self._workers:
            await self._queue.put(None)
        await asyncio.wait_for(
            asyncio.gather(*self._workers, return_exceptions=True),
            timeout=timeout,
        )
        self._workers.clear()

    async def submit(self, task: AgentTask) -> None:
        """Enqueue a task. Blocks (backpressure) if the queue is full."""
        if self._max_queue is not None:
            await self._queue.put(task)
        else:
            await self._queue.put(task)

    async def _worker_loop(self, worker_id: int) -> None:
        while True:
            task = await self._queue.get()
            try:
                if task is None:
                    return
                self._active += 1
                try:
                    result = await asyncio.wait_for(
                        self._executor(task),
                        timeout=task.timeout_s,
                    )
                    self._completed += 1
                except asyncio.TimeoutError:
                    self._failed += 1
                except Exception:
                    self._failed += 1
                finally:
                    self._active -= 1
            finally:
                self._queue.task_done()

This implementation is deliberately simple, and it is correct for the common case. But note two things it does not yet do: it does not return results to callers, and it does not apply rate limits or circuit breakers. We will add those.

Why a semaphore alone is not enough

You may be tempted to write this instead:

python
sem = asyncio.Semaphore(20)

async def guarded(task):
    async with sem:
        return await run_agent(task)

This caps concurrency, which is better than unbounded fan-out. It is still not a worker pool, for three reasons.

  1. No bounded queue. Callers still create every task up front. If you submit 100,000 tasks, you have 100,000 tasks in memory, each holding a payload. The semaphore limits execution, not admission.
  2. No natural priority or ordering. Priority requires a priority queue or a scheduler, not a semaphore.
  3. No lifecycle. There is no clean start, drain, or shutdown path, and no place to attach pool-level metrics.

The semaphore pattern is fine for a bounded fan-out of a few hundred items. For a fleet, use an explicit queue and workers.

3. Building a Bounded Worker Pool

Let's build a pool that returns results, supports priority, and exposes honest metrics. This is the version you would actually deploy.

python
import asyncio
import heapq
import itertools
import time
from dataclasses import dataclass, field
from typing import Any


@dataclass(order=True)
class _QueueItem:
    sort_key: tuple[int, int] = field(compare=True)
    task: AgentTask = field(compare=False)


@dataclass
class PoolStats:
    submitted: int = 0
    completed: int = 0
    failed: int = 0
    rejected: int = 0
    active: int = 0
    queued: int = 0
    total_wait_ms: int = 0


class PriorityWorkerPool:
    def __init__(
        self,
        executor: Executor,
        *,
        max_workers: int = 8,
        max_queue: int = 1000,
        name: str = "agent-pool",
    ) -> None:
        self._executor = executor
        self._max_workers = max_workers
        self._max_queue = max_queue
        self._name = name
        self._heap: list[_QueueItem] = []
        self._counter = itertools.count()
        self._condition = asyncio.Condition()
        self._results: dict[str, asyncio.Future[AgentResult]] = {}
        self._workers: list[asyncio.Task[None]] = []
        self._stats = PoolStats()
        self._closed = False

    async def start(self) -> None:
        if self._workers:
            return
        self._workers = [
            asyncio.create_task(self._worker_loop(i), name=f"{self._name}-{i}")
            for i in range(self._max_workers)
        ]

    async def submit(self, task: AgentTask) -> asyncio.Future[AgentResult]:
        """Admit a task or raise QueueFull.

        Raises `asyncio.QueueFull` when the backlog exceeds max_queue.
        """
        if self._closed:
            raise RuntimeError("pool is closed")
        async with self._condition:
            if len(self._heap) >= self._max_queue:
                self._stats.rejected += 1
                raise asyncio.QueueFull(
                    f"{self._name}: queue full ({len(self._heap)}/{self._max_queue})"
                )
            future: asyncio.Future[AgentResult] = asyncio.get_running_loop().create_future()
            self._results[task.task_id] = future
            # Higher priority value sorts first; FIFO within a priority band.
            heapq.heappush(
                self._heap,
                _QueueItem(sort_key=(-task.priority, next(self._counter)), task=task),
            )
            self._stats.submitted += 1
            self._stats.queued = len(self._heap)
            self._condition.notify()
            return future

    async def _worker_loop(self, worker_id: int) -> None:
        while True:
            async with self._condition:
                while not self._heap and not self._closed:
                    await self._condition.wait()
                if self._closed and not self._heap:
                    return
                item = heapq.heappop(self._heap)
                self._stats.queued = len(self._heap)

            task = item.task
            started = time.perf_counter()
            self._stats.active += 1
            try:
                result = await asyncio.wait_for(
                    self._executor(task), timeout=task.timeout_s
                )
                self._stats.completed += 1
            except asyncio.TimeoutError:
                result = AgentResult(
                    task_id=task.task_id,
                    ok=False,
                    error=f"timeout after {task.timeout_s}s",
                )
                self._stats.failed += 1
            except Exception as exc:
                result = AgentResult(
                    task_id=task.task_id,
                    ok=False,
                    error=f"{type(exc).__name__}: {exc}\",