Agentic Inference: A Practical Guide to KV Cache, Scheduling, and Throughput

A tool-using agent can be idle and expensive at the same time.

The model emits a tool call, the harness starts a shell command or API request, and generation pauses. From the application’s perspective, useful work is happening. From the inference server’s perspective, the request has disappeared. Its attention state may remain in scarce GPU memory, move to a slower tier, or be destroyed and recomputed when the agent returns.

That decision is repeated at every tool boundary. Make it badly and the system pays in one of three currencies:

• GPU memory held by work that cannot currently run;
• transfer time to restore cached state;
• prefill compute to rebuild state that was discarded.

This is why agentic inference is not ordinary chat serving with more requests. It is a stateful scheduling problem whose useful unit is the trajectory—the whole model/tool/model program—not the individual API call.

This guide develops that argument from first principles, turns it into an implementation plan, and gives you an experiment you can run against your own workloads. It was prompted by Joel Smith’s careful field study, Improving Throughput by Optimising KV Cache Efficiency for Agentic Workloads, and cross-checked against the systems literature and current vLLM documentation. Smith’s measurements are especially useful because the article reports its limitations plainly: the runs were expensive, trajectories varied between runs, conditions were not repeatedly sampled, and the resulting throughput uplift is directional rather than statistically established. We will preserve that boundary throughout.

The workload changed before the serving API did

A chat request and an agent turn can look identical at an OpenAI-compatible endpoint. Both contain messages, sampling parameters, and perhaps tool schemas. The difference is outside the request.

A harness knows that:

• the current response is one step in a longer program;
• a specific tool is running;
• some tools usually finish in milliseconds and others in seconds;
• a parent may be waiting on several subagents;
• one branch is on the critical path while another is speculative;
• the trajectory has finished, failed, or been cancelled;
• the next turn will probably reuse almost the entire existing prefix.

A request-oriented inference server usually sees none of this. It receives one turn, returns a tool call, and later receives another request containing the expanded transcript. The API boundary throws away exactly the state needed to decide whether the trajectory’s KV cache should stay resident.

The workload also has a distinctive shape. Agent prompts contain system instructions, tool definitions, repository or document context, prior reasoning, and tool results. The prefix grows across many turns, while the generated answer at each turn may be comparatively short. Smith’s SWE-bench Lite run, for example, observed a median of 15,146 prefill tokens and 501 decode tokens per turn. Treat those as one workload sample, not a universal ratio. The architectural implication is durable: repeated prefill can dominate the avoidable work in long, multi-turn trajectories.

Research reached the same conclusion from several directions:

These figures are not directly comparable. The papers use different models, hardware, arrival processes, baselines, objectives, and workload graphs. Their shared result is more important than any maximum: application state is a useful serving signal.

What the KV cache stores—and why it becomes the bottleneck

During transformer inference, every layer produces key and value vectors for each processed token. Decode reuses those vectors instead of recomputing the entire prefix for every new token. The stored tensors are the KV cache.

For a conventional attention layer, a useful first-order estimate is:

bytes_per_token = 2
                × number_of_layers
                × number_of_KV_heads
                × head_dimension
                × bytes_per_element

The leading 2 represents keys and values. Total sequence storage is approximately:

KV_bytes ≈ bytes_per_token × cached_tokens × parallelism_adjustment

The adjustment matters: tensor parallelism may shard KV state, while replication and some attention implementations change what each device holds. Allocator metadata, block rounding, hybrid attention, and alignment add details. Use the equation for capacity planning, then verify it against runtime metrics.

Smith’s Qwen3-32B example has 64 layers, 8 KV heads, head dimension 128, and BF16 elements:

2 × 64 × 8 × 128 × 2 bytes = 262,144 bytes/token = 256 KiB/token

A 15,146-token prefix is therefore about 3.7 GiB before implementation-specific overhead. One paused trajectory can occupy gigabytes of high-bandwidth memory.

The foundational systems result is PagedAttention. Instead of reserving one contiguous maximum-size region per sequence, vLLM divides KV storage into blocks and maps logical sequence blocks to physical GPU blocks. This sharply reduces fragmentation and enables sharing. It does not answer the next question: which blocks deserve scarce residency when many paused and runnable programs compete?

Prefix caching is necessary but not sufficient

Automatic prefix caching hashes completed KV blocks and reuses them when a later request has an identical token prefix. The vLLM design uses chained block hashes so a block’s identity includes its preceding prefix. SGLang’s RadixAttention represents reusable prefixes in a radix tree and reported large throughput gains on structured language-model programs.

Prefix caching changes a returning turn from this:

tokenize full transcript → prefill full transcript → decode next action

to this:

match cached prefix → prefill only the appended delta → decode next action

But a cache lookup only helps when the blocks still exist somewhere useful. Under pressure, an engine must choose among three actions:

1. Retain in GPU memory. Resume quickly, but deny that memory to runnable work.
2. Offload to CPU RAM or another tier. Free GPU capacity, but pay transfer and allocation costs on return.
3. Discard. Free capacity completely, but pay recomputation on return.

Plain LRU considers the past: “Which block was used least recently?” An agent harness often has partial knowledge of the future: “The file edit normally returns in 90 ms,” “this database query has a one-second p90,” “the parent cannot continue until this child finishes,” or “this trajectory is over.” That makes recency an incomplete eviction signal.

The economic decision: retain, reload, or recompute

Do not turn the choice into a universal timeout. Estimate the cost for the current sequence and current system state.

Let:

S  = KV state size in bytes
G  = predicted tool-gap duration in seconds
B  = effective reload bandwidth in bytes/second
A  = destination-allocation and queue overhead in seconds
P  = measured prefill time to rebuild the reusable prefix
M  = opportunity cost per GPU-byte-second under current load

Then a simplified comparison is:

retain_cost    ≈ S × G × M
reload_time    ≈ S / B + A
recompute_time ≈ P

Offload is attractive when reload is faster than recomputation and the freed capacity can do useful work during the gap. Retention is attractive when the gap is so short that eviction and restoration consume more capacity-time than they release. Discard is reasonable when reuse is unlikely, the trajectory is complete, the cache is incompatible with the future route, or recomputation is cheaper than a slow-tier read.

The hard part is M. Under light load, free memory has little opportunity cost; retaining state may be harmless. Near saturation, the same retained gigabyte may block an entire runnable sequence. The policy therefore needs load, not just tool duration.

A practical first policy can be deliberately coarse:

The last row is important. Hints should degrade to normal serving behavior, not become correctness requirements.

A buildable architecture

You do not need to begin with a custom scheduler fork. Start with a thin state bridge between the harness and the serving layer.

agent harness
    │ emits lifecycle + tool-state events
    ▼
inference gateway / state bridge
    ├── trajectory registry
    ├── tool-duration estimates
    ├── prefix-affinity router
    └── cache-advice policy
            │
            ├── request priority / routing key
            ├── retain, offload, prefetch, release hints
            └── observability events
                    ▼
             inference replicas
             GPU KV ↔ CPU KV ↔ remote/disk tier

The following is a non-normative deployment profile, not a change to the ContextOS runtime contract:

type TrajectoryEvent = {
  trajectoryId: string
  turnId: string
  event:
    | "model_started"
    | "tool_started"
    | "tool_finished"
    | "trajectory_finished"
    | "trajectory_cancelled"
  toolClass?: string
  emittedAtMs: number
  expectedNextAgent?: string
  criticalPath?: boolean
}
 
type CacheAdvice = {
  trajectoryId: string
  action: "default" | "retain" | "offload" | "prefetch" | "release"
  ttlMs?: number
  priority?: number
  reason:
    | "short_tool_gap"
    | "capacity_pressure"
    | "next_step_known"
    | "terminal_state"
    | "insufficient_evidence"
  policyVersion: string
}

Keep the event contract small. Do not send raw chain-of-thought, tool secrets, or full prompts to the scheduler. It needs lifecycle and cost signals, not semantic access to private working data.

Step 1: make prefixes reusable

Before changing eviction, make cache identity stable:

• keep the system prompt, tool schemas, and shared reference context at the beginning;
• put volatile user input, retrieved evidence, tool output, and timestamps later;
• serialize tools and JSON schemas deterministically;
• pin tokenizer, chat template, model revision, adapter identity, and cache format;
• route a trajectory back to a compatible model replica;
• measure the matched prefix length, not merely “cache hit: yes.”

A cache miss caused by nondeterministic tool order cannot be repaired by a smarter eviction algorithm.

Step 2: turn on block-level prefix caching

On current vLLM releases, automatic prefix caching can be enabled through the engine configuration. Exact flags and compatibility change, so use the documentation for the version you deploy. Confirm four behaviors with a replayed prompt pair:

1. the second request matches the expected number of prefix tokens;
2. output quality and tokenization are unchanged;
3. the cache is invalidated when model or adapter identity changes;
4. observability distinguishes GPU-local, offloaded, and recomputed tokens.

Prefix caching reduces prefill work. It does not automatically preserve a paused trajectory under memory pressure.

Step 3: evaluate KV precision

KV quantization reduces bytes per token. Moving BF16 KV to FP8 roughly halves raw KV storage for compatible models and hardware. That can raise the concurrency point at which eviction begins and can reduce memory traffic during decode.

Do not assume “half the bytes” means “twice the throughput.” Kernel support, scale calibration, attention backend, model quality, context length, and hardware FP8 capability all matter. Run task-level quality evals and measure decode throughput, TTFT, and completed trajectories. Keep model-weight quantization and KV-cache quantization as separate experiment factors.

Step 4: add a larger cache tier

The current vLLM KV offloading guide describes completed GPU blocks copied to pinned host memory and promoted on demand, with asynchronous GPU–CPU transfer. Systems such as LMCache extend sharing across engines and storage tiers. Mooncake goes further: its production architecture treats distributed KV state across GPU, CPU, SSD, and network resources as a first-class cluster service.

Size the tier from traces:

required_offload_capacity
  ≈ concurrent_paused_trajectories
  × p90_KV_bytes_per_trajectory
  × headroom_factor

Then verify the actual transfer curve. Small random block movement, NUMA placement, PCIe topology, pinned-memory limits, competing transfers, and destination allocation can leave realized bandwidth far below a link’s headline number.

Step 5: export lifecycle hints

Start with the safest high-value signal: terminal state. When the harness knows a trajectory is done, the serving layer should not wait for recency to discover that its blocks are useless.

Next, estimate gaps by tool class. An exponentially weighted moving average is a reasonable baseline:

estimate_next = α × observed_gap + (1 - α) × estimate_previous

Use quantiles when the distribution has a long tail. A mean can make a 20 ms fast path and a 20 s slow path look like one mediocre tool. Partition by the smallest stable class that has enough samples: tool type, route, perhaps operation class. Avoid per-trajectory models until the data shows they help.

Convert the estimate into expiring advice. A retain decision without a TTL can turn one stalled tool into a persistent memory leak.

Step 6: schedule programs, not only calls

Cache policy and request scheduling interact. Protecting a prefix is wasted if the returning request sits at the back of the admission queue until its protection expires. Conversely, prioritizing every resumption can starve new work.

Program-level schedulers address this by accounting for cumulative service across the trajectory. Autellix, for example, applies program-aware least-attained-service ideas so a program that has received little total service can advance without letting a chain of individually short calls monopolize the engine.

A production policy usually needs all four concerns:

• fairness: bound how long cold arrivals and large programs can wait;
• locality: prefer work whose prefix is already resident when the fairness cost is acceptable;
• critical path: advance steps that unblock the workflow outcome;
• SLO class: preserve explicit product priorities without allowing unbounded priority inflation.

Record the reason for every priority change. Otherwise a throughput optimization becomes an unexplainable tenant-isolation problem.

Step 7: prefetch when the graph gives you an oracle

Multi-agent workflows often expose a partial execution graph. If the next agent or join is known, begin loading its prefix before it becomes runnable. KVFlow formalizes this as “steps to execution”: entries closer to future use survive eviction, and the next step’s state is prefetched concurrently with current work.

Only prefetch when:

P(next_step) × avoided_stall
  > transfer_cost + pollution_cost + wrong-path cost

Data-dependent branches weaken the oracle. Keep a confidence threshold, cap speculative bytes, and cancel prefetch when the branch changes.

Observability: measure trajectories, not token counters alone

Traditional serving dashboards can improve while users wait longer. Higher batch token throughput may coincide with worse job completion time if paused trajectories are repeatedly evicted or resumptions starve.

Collect at least these dimensions:

Two ratios are particularly revealing:

prefix_reuse_ratio = reused_prefix_tokens / eligible_prefix_tokens
 
rebuild_tax = recomputed_prefix_tokens / total_prefill_tokens

Split reuse into GPU-local and lower-tier hits. A single “cache hit rate” hides whether the system avoided compute but added a long transfer stall.

Use Little’s law carefully:

throughput X = average_concurrency L / average_time_in_system W

It is useful for steady-state sanity checks. It does not rescue a closed-loop benchmark from coordinated omission or prove causality in a noisy run.

A benchmark protocol you can trust

Agent benchmarks are harder than prompt benchmarks because model outputs change the future workload. One run may use three tools and another fifteen. Batch composition can also change floating-point results, which changes tool choices and trajectory length. If you compare raw wall time without controlling useful work, scheduler noise can masquerade as intelligence or vice versa.

Use this protocol.

1. Capture a workload manifest

Pin and record:

• model, revision, tokenizer, chat template, quantization, and attention backend;
• runtime version and commit;
• GPU model, count, topology, driver, and host-memory layout;
• prompt and tool-schema hashes;
• task dataset revision and sandbox image;
• arrival process, concurrency, random seeds, and maximum context;
• offload capacity, block size, scheduler policy, and all cache flags.

2. Separate replay from live-agent evaluation

Run both:

Trace replay fixes request lengths, arrival times, tool gaps, and prefix relationships. It isolates serving behavior and supports repeated trials.

Live trajectories preserve the feedback loop between model output and future calls. They measure real task outcomes but require more repetitions and outcome normalization.

Do not use live trajectories alone to attribute a small systems uplift.

3. Use an open-loop arrival process

A fixed number of synchronous workers lowers offered load when the server slows down. That hides overload. For capacity curves, submit arrivals independently—often with a Poisson process or replayed production timestamps—and measure what the service actually sustains.

Closed-loop concurrency remains useful for saturation sweeps and cost control, but label it accurately.

4. Sweep through the memory elbow

Test light load, the onset of KV pressure, and overload. The optimization may do nothing under light load and become decisive only once blocks churn. For each condition, report:

• offered and achieved trajectory rate;
• p50, p95, and p99 job completion time;
• task success or accepted-outcome rate;
• local, offloaded, and missed prefix-token ratios;
• recomputed tokens;
• bytes transferred and achieved bandwidth;
• GPU KV occupancy and preemption count.

5. Use ablations, not a single “optimized” arm

At minimum:

A: baseline, no prefix caching
B: prefix caching
C: B + KV quantization
D: B + offload
E: B + lifecycle hints
F: B + program-aware scheduling
G: combined policy

This tells you whether the gain came from extra capacity, fewer prefills, better routing, faster transfers, or state-aware decisions.

6. Repeat and report uncertainty

Run enough independent trials to show confidence intervals or, at minimum, the distribution and sample count. Randomize condition order to reduce thermal, network, and neighbor effects. When live trajectories complete different work, normalize with accepted outcomes and stratify by task difficulty.

Smith’s experiment saw a 1.04–1.27× apparent throughput uplift from state hints across tested concurrency points, alongside higher GPU-local prefix hits. The article also observed a transfer-bandwidth confound and run-to-run trajectory variation, and explicitly concluded that controlled repetitions were needed. That is the right reporting standard: the result motivates an experiment; it does not close the question.

A safe rollout sequence

The operational path should be reversible.

1. Observe only. Add trajectory IDs, lifecycle events, prefix-token accounting, and tool-gap distributions. Do not change scheduling.
2. Stabilize prefixes. Fix nondeterministic serialization and routing. Enable prefix caching and verify reuse.
3. Test precision. Evaluate FP8 or another supported KV format behind a model-specific flag with task-quality gates.
4. Add offload. Start with bounded CPU capacity; alert on transfer queueing and host-memory pressure.
5. Release terminal state. Reclaim completed and cancelled trajectories immediately.
6. Shadow advice. Compute retain/offload/prefetch choices without applying them; compare predictions with actual return times.
7. Canary hints. Apply advice to a small traffic slice with TTLs, byte caps, and an instant fallback to default policy.
8. Canary program scheduling. Enforce fairness and per-tenant budgets before increasing the slice.
9. Replay before promotion. Re-run representative traces and live tasks; promote only when outcome, latency, fairness, and quality gates all pass.

Do not ship a throughput win that silently worsens p99 completion time, task success, or tenant fairness.

Failure modes worth designing for

Cache identity drift

A model revision, tokenizer, RoPE configuration, LoRA adapter, multimodal input, or chat-template change can make old KV state incompatible. Include every compatibility dimension in the cache namespace. Fail closed on ambiguity.

Hint races

If advice arrives after blocks enter eviction, “retain” cannot resurrect them. Emit lifecycle state before or atomically with the transition where possible, measure hint-application lag, and treat late hints as a first-class metric.

Prediction tails

An EWMA can retain state for a tool that occasionally stalls for minutes. Every retain action needs a TTL and a global pressure override.

Offload cache churn

A small CPU tier can cycle through blocks just as a small GPU tier does. Capacity planning must include paused concurrency and trajectory length, not just average prompt size.

Transfer storms

If many tools finish together, synchronized reloads can saturate PCIe or the network. Bound restore concurrency, prioritize critical paths, and add jitter or admission control where appropriate.

Locality-induced starvation

Always selecting cache-hot work can indefinitely defer cache-cold requests. Combine locality with aging or attained-service fairness.

Speculative pollution

Prefetching both sides of an uncertain branch can evict state for work that is definitely runnable. Budget speculative bytes and measure how much prefetched state is consumed before eviction.

Privacy and deletion

KV tensors are derived from prompts and should inherit their data classification. Define tenant isolation, encryption and process boundaries, retention, deletion, crash cleanup, and incident handling for every storage tier. A “cache” is still stored customer-derived state.

How to choose the next optimization

Use evidence from the bottleneck:

The sequence matters. A custom eviction policy cannot recover a prefix that changes every turn. An enormous offload tier cannot compensate for a queue that ignores resumptions. More GPUs can hide a bad policy while multiplying its cost.

The larger systems lesson

PagedAttention borrowed virtual-memory ideas for KV allocation. Agent-aware inference completes the analogy.

An agent trajectory resembles a program. Model turns are bursts of accelerator execution. Tool calls are I/O waits. KV blocks are a working set. The harness knows process state; the inference engine owns scarce compute and memory. Efficient execution requires a narrow contract between them.

The contract should not expose private reasoning or make the model responsible for resource policy. It should expose deterministic lifecycle facts and measured predictions: this trajectory paused on this tool, this branch is likely next, this work is complete, this prefix is compatible, this SLO applies. The serving layer can then make an auditable placement and scheduling decision.

That is the durable idea behind the recent work. The KV cache is not merely an attention optimization. In an agent system, it is execution state—and execution state becomes far more valuable when the scheduler understands the program that will use it.

Research sources

• Joel Smith: Improving Throughput by Optimising KV Cache Efficiency for Agentic Workloads — motivating field experiment, vLLM scheduler walkthrough, state-hint prototype, and explicit limitations.
• PagedAttention / vLLM — block-based KV memory management and sharing.
• vLLM automatic prefix caching design — block hashing, allocation, cache lookup, and eviction mechanics.
• vLLM KV offloading guide — current multi-tier offload behavior and configuration surface.
• SGLang — RadixAttention and structured language-model program execution.
• InferCept — adaptive handling of inference interruptions caused by tools and external interactions.
• Preble — distributed scheduling that co-optimizes prefix reuse and load balance.
• Mooncake — production, disaggregated, KV-centric serving across a storage hierarchy.
• Autellix — program-level scheduling for LLM agents.
• KVFlow — workflow-aware eviction and prefetch for multi-agent systems.
• Continuum — tool-aware cache TTL and continuity-aware scheduling.
• TokenCake — cache-centric scheduling around multi-agent function-call stalls.
• LMCache — reusable KV storage and sharing across inference engines and queries.

What to read next

• LLM Latency Engineering: TTFT, Caching, Routing, and the Road to Real-Time Agents
• Graph Engineering for AI Agent Workflows
• Orchestration
• Evaluation and Observability