LLM Latency Engineering: TTFT, Caching, Routing, and the Road to Real-Time Agents

“The model returns 200 tokens per second” is not a latency architecture.

A user experiences queueing, network travel, input processing, the wait for the first useful token, the cadence of later tokens, tool calls, validation, retries, and UI rendering. An operator experiences p95 and p99 under bursts, cold starts during deploys, cache misses after prompt edits, regional failures, and cost when traffic must move to a premium tier.

The production objective is therefore not maximum tokens per second. It is:

Minimize time and cost to an accepted outcome while keeping quality, policy, evidence, and reliability inside their declared bounds.

The distinction matters more after the last ten days of releases. OpenAI announced a limited Ultrafast mode at up to 750 output tokens per second and 14 times Standard for GPT-5.6 Sol, while Fast mode expanded to prompts above 272K tokens. SGLang 0.5.17 and vLLM 0.27 shipped work across frontends, prefix caches, prefill, speculative decode, cold starts, fault tolerance, and disaggregation.

Those developments raise the ceiling. They do not remove the need to identify the critical path.

Start with the latency equation

For a single model turn, a useful decomposition is:

T_model_turn = T_admission + T_queue + T_network + T_tokenize
             + T_prefill + T_first_decode
             + Σ T_later_decode + T_stream_and_parse

For a tool-using workflow:

T_accepted_outcome = Σ T_model_turn
                   + Σ T_tool
                   + T_validation
                   + T_retry_or_recovery
                   + T_human_wait_when_required

The terms interact. A shorter output reduces sequential decode work. A stable prefix can reduce prefill. Parallel calls can reduce wall time and increase contention. A smaller model can decode faster and require more retries. A premium service tier can accelerate model time and leave a slow tool unchanged.

Never optimize a term before confirming it is material in the traces that violate the SLO.

The metrics that answer different questions

Record distributions by route and slice. “p95 latency” without model, service tier, prompt version, input/output lengths, cache state, concurrency, region, and tool path is not actionable.

A span model that makes slow runs explainable

One trace should be able to answer where time went.

type ModelSpan = {
  traceId: string
  routeId: string
  model: string
  serviceTier: string
  promptVersion: string
  stablePrefixHash: string
  inputTokens: number
  cachedInputTokens: number
  cacheWriteTokens?: number
  reasoningTokens?: number
  outputTokens: number
  queueMs: number
  ttftMs: number
  decodeMs: number
  totalModelMs: number
  cacheState: "hit" | "partial" | "miss" | "ineligible"
  coldStart: boolean
  accepted: boolean
  failureClass?: string
}

Add child spans for retrieval, tools, validation, retries, and UI delivery. Preserve the exact runtime versions needed to compare a slow run with its baseline.

The optimization ladder

Work down the ladder in roughly this order. The exact priority depends on the workload, but application-level changes are usually easier to validate and reverse than kernel or cluster changes.

1. Do not call a generative model when a deterministic path is enough

OpenAI’s latency guide ends with a blunt principle: do not default to an LLM. Exact lookup, a parser, a rules engine, a cached artifact, or a small classifier can be faster and more reliable for bounded decisions.

Examples:

• validate an identifier with code, not a prompt;
• compute arithmetic in a calculator or code tool;
• reuse an approved static answer when the source version and request intent match;
• check hard policy thresholds in a deterministic engine;
• return a form when the next step is known.

This is not anti-model design. It keeps probabilistic reasoning for the places that require it.

2. Generate fewer sequential tokens

Autoregressive output is sequential. OpenAI’s current guide gives a directional heuristic that halving output length may roughly halve latency, and calls generation the highest-latency step in most ordinary workloads. The exact gain depends on model, route, reasoning, batching, and application behavior, but the priority is sound.

Use four controls:

1. Output contract. Ask for the fields or prose actually used.
2. Verbosity or length setting. Prefer API controls when available and add task-specific limits in the prompt.
3. Stop and maximum-output limits. Treat them as safety rails, not a substitute for a good contract.
4. Reasoning budget. Select model effort by route; more internal work can raise latency even when visible output is short.

Bad:

Analyze the ticket in great detail. Explain all your reasoning and provide a comprehensive answer.

Better:

Return the disposition, up to three supporting evidence references, the missing fact if any,
and a customer response under 120 words. Omit internal deliberation.

Do not cut evidence, caveats, or required steps simply to win a latency chart. The measure is accepted outcome, not raw brevity.

3. Remove unnecessary serial turns

Every serial request adds round-trip, queue, prefill, and decode time. Combine steps when one model can safely return several named fields. Use programmatic tool orchestration when a bounded stage can call, filter, join, or aggregate several tools without new semantic judgment after each result.

Keep turns separate when:

• a result changes the next decision materially;
• a side effect requires approval;
• a validator must stop the workflow;
• native citations or artifacts would be lost;
• combining tasks makes the prompt less reliable.

“One call” is not automatically better. A giant call that mixes extraction, policy judgment, tool planning, and final prose may retry more often than a well-designed two-stage route.

4. Parallelize only independent work

Parallel retrievals or model calls can replace a sum of durations with approximately the maximum duration plus coordination overhead.

serial:   T = A + B + C
parallel: T ≈ max(A, B, C) + coordination

Parallelism is appropriate when tasks have no dependency and do not contend over mutable state. It can worsen tail latency through queueing, rate limits, memory pressure, and fan-out amplification.

Bound concurrency. Set per-branch deadlines. Cancel obsolete work. Do not retry every branch at once. Track useful fan-out: how many parallel results changed the accepted outcome?

5. Design the prompt for prefix reuse

Prompt caching can lower prefill time and cost when a long initial segment repeats. Provider details differ, but the stable-prefix pattern is consistent.

stable:   policy → tools → schema → examples → shared reference
volatile: retrieved evidence → session delta → user input → current task

OpenAI’s current GPT-5.6 cache controls expose breakpoints, a cache key, cache reads, and cache writes. The guide says:

• exact prompt-prefix matching is required through the breakpoint;
• the eligible prefix must meet a model-specific minimum;
• changing tools, schemas, images, or content before the breakpoint can invalidate reuse;
• cached_tokens and cache_write_tokens should be monitored together;
• a stable cache key helps related requests reach matching cache state;
• high traffic on one key may reduce hit rate, so larger workloads need deterministic partitioning.

Anthropic’s prompt caching guide similarly includes tools, system content, and messages in the prefix and requires exact matching. Google’s implicit caching guide recommends common content first and similar requests close in time.

Cache economics

Let:

• U be uncached input-token cost;
• W be cache-write cost;
• R be cache-read cost;
• h be the probability a written prefix is reused;
• n be expected reads after one write.

A simplified cached-prefix cost is:

C_cached = W + nR
C_uncached = (n + 1)U

Caching is economically useful when W + nR < (n + 1)U, but cost is only one part of the decision. Cache reuse can reduce prefill and TTFT; storing or routing cache state can affect privacy, residency, affinity, and failure behavior. Use current provider rates and data-control terms at decision time.

Cache failure modes

• a timestamp in the system prompt causes a miss every request;
• dynamic tool order changes the rendered prefix;
• a prompt optimizer edits examples faster than the cache can be reused;
• round-robin routing scatters a hot session across replicas without shared KV state;
• one cache key becomes a high-volume hotspot;
• a team celebrates hit rate while low-quality stale context remains cached.

Cache correctness includes source version and invalidation, not only byte identity.

6. Route by task, difficulty, and latency budget

Model routing is usually a larger lever than polishing a few prompt tokens. Define routes by evaluated task class.

Route inputs should be deterministic or versioned: intent, risk, prompt length, expected output, tool path, SLO class, and any approved customer tier. Do not route based on a random draw or an unrecorded model guess.

type RouteInput = {
  intent: "classify" | "answer" | "investigate"
  risk: "low" | "high"
  latencyClass: "interactive" | "standard" | "offline"
  expectedOutputTokens: number
}
 
function selectRoute(input: RouteInput) {
  if (input.intent === "classify" && input.risk === "low") {
    return { model: "fast-small", reasoning: "none", tier: "standard" }
  }
  if (input.latencyClass === "interactive" && input.risk === "low") {
    return { model: "balanced", reasoning: "low", tier: "fast" }
  }
  if (input.risk === "high" || input.intent === "investigate") {
    return { model: "frontier", reasoning: "high", tier: "standard" }
  }
  return { model: "balanced", reasoning: "low", tier: "standard" }
}

The actual model ids belong in configuration, not evergreen logic. Every route needs a fallback, timeout, retry policy, and promotion record.

7. Use premium service tiers where model time is the critical path

OpenAI’s August 2026 updates illustrate two premium routes:

• Fast mode supports long-context GPT-5.6 requests and is advertised at up to 2.5 times Standard;
• Ultrafast is a limited preview for GPT-5.6 Sol, powered by Cerebras, advertised at up to 750 output tokens per second and 14 times Standard.

Benchmark them with the same prompt, tools, region, dataset, concurrency, and validators. Capture the actual service tier returned by the API where applicable. Measure:

• TTFT and ITL at p50/p95/p99;
• downgrade or capacity behavior;
• errors and retries;
• accepted-outcome rate;
• cost per accepted outcome;
• conversion or task-completion effect.

A premium tier is wasted when retrieval or a human approval dominates the path. It can be valuable when fast generation changes an interactive loop from turn-taking to flow.

8. Improve perceived latency without lying about completion

Streaming, optimistic UI, progressive retrieval, cached previews, and early status messages can reduce waiting pain. They do not make the underlying work complete sooner.

Good patterns:

• stream verified prose when later content cannot invalidate it;
• show which stage is running for a long tool workflow;
• render sources as they are validated;
• prefetch likely read-only context after a stable intent signal;
• allow cancellation and preserve resumable state.

Bad patterns:

• show unvalidated text that may reverse a consequential decision;
• label “model finished” as “task completed” before a tool postcondition passes;
• animate progress with no connection to real stages;
• stream hidden policy or private reasoning.

Track time to first useful output and true time to accepted outcome separately.

9. For self-hosting, optimize the phase that is saturated

Self-hosted inference adds powerful controls and operational responsibility. The main techniques attack different bottlenecks.

What the August releases add

SGLang 0.5.17 adds an initial Rust frontend, session-aware radix caching, a prefill strategy for MoE, faster recovery and loading paths, and reduced host work around speculative decode. The release marks at least one prefill feature early-development and gates several optimizations behind flags.

vLLM 0.27.0 adds JIT warmup work, several model-specific TTFT and kernel improvements, more speculative-decoding support, hybrid-model P/D disaggregation, fault-tolerance work, and an integrated benchmark path. The 0.27.1 patch adds quantized DSpark Markov heads.

Treat each release as a candidate, not a command to upgrade globally. Reproduce on your hardware, model, prompt/output distribution, concurrency, and SLO.

10. Benchmark production shape, not a warm toy request

A useful benchmark matrix varies the factors that change the bottleneck.

The August 6 preprint LLM Inference Under Bursty Workload Distribution is simulation-based and does not prove one scheduler is best. It does highlight the mistake of assuming constant-rate Poisson traffic. Replay observed arrival patterns and autocorrelation from production.

The August 4 LLM Serving in the Wild study reports limited multi-framework use in its open-source dataset. That is another operational clue: framework migrations have substantial integration and learning costs. Optimize within the current stack first unless a representative benchmark shows a durable advantage large enough to repay the move.

Speculative decoding: exact, approximate, and workload-dependent

Speculative decoding uses a cheaper draft process to propose tokens and a target model to verify several in parallel. Speedup depends on:

• draft speed;
• accepted tokens per verification step;
• target verification cost;
• batch size and concurrency;
• prompt and output domain;
• model architecture and hardware;
• compatibility with tools, structured outputs, quantization, and attention modes.

The August 4 preprint Approximate Speculative Decoding reports a 3.05%–15.26% fixed-workload throughput gain over matched strict verification across its seven Qwen3-14B plus DSpark-14B tasks, averaging 7.78%. Its nonzero regret budget can accept selected mismatches, so it may change the generated trajectory.

That makes route policy essential:

exact verification:
  default for consequential decisions, code changes, tool arguments, and strict schemas
 
approximate verification:
  eligible only for evaluated low-risk routes
  fixed regret and exception budgets
  zero-budget fallback
  quality and policy monitoring by slice

Do not infer application speedup from verifier acceptance alone. Measure the entire accepted-outcome path.

The latency scorecard

Every optimization candidate should produce one reviewable card.

A candidate is not “faster” if it drops difficult cases, times out less visibly, produces shorter incomplete answers, or shifts latency into a human review queue.

Common optimization mistakes

Optimizing average tokens per second

The median improves while p99 queueing violates the SLO. Track the stage distributions under production-shaped load.

Shortening the wrong side

A team spends weeks reducing a 3K-token prompt by 20% while the model emits 2K tokens. Start from stage time, not token ideology.

Streaming as the only fix

The UI feels faster, but tools and validation still finish late. Keep true completion visible.

Cache hit rate without cache value

A high hit rate on a small prefix may save little. A stale or over-broad shared prefix can harm quality. Track tokens and accepted outcomes.

Fan-out without a budget

Parallel agents reduce one run’s wall time and multiply system load, rate limits, and tail risk. Bound concurrency and cancel losing branches.

Speculation by default

Acceptance is poor on the actual domain or a new tool/schema path is incompatible. Gate it by route and evaluator.

Benchmarking only warm steady state

Deploys, autoscaling, compilation, model loading, and cache loss define real incidents. Include them.

Premium tier everywhere

Spend rises where the critical path is retrieval or tools. Route premium capacity only where model latency changes the outcome.

A 90-day roadmap

Days 0–30: instrument and classify

• Define SLOs for interactive, standard, quality-first, and offline routes.
• Trace queue, prefill/TTFT, decode, tools, validation, retries, and true completion.
• Record input, cached input, cache writes, reasoning, and output tokens.
• Build prompt/output/concurrency distributions and a burst replay trace.
• Inventory serial model calls, fan-out, cache boundaries, and cold paths.

Exit condition: every slow accepted or failed run can be attributed to a stage, slice, and versioned route.

Days 31–60: improve the application path

• Remove unnecessary model calls.
• Set output and reasoning budgets by route.
• Collapse redundant serial steps and parallelize bounded independent work.
• Stabilize reusable prefixes and monitor cache reads and writes.
• Route simple work to a smaller model and offline work away from interactive capacity.
• Add streaming and progress only where partial results are safe.

Exit condition: p95 accepted-outcome latency improves with quality and policy inside the declared gate.

Days 61–90: tune capacity and serving

• Compare provider service tiers on the same scorecard.
• If self-hosting, profile queue, prefill, decode, communication, and host overhead separately.
• Canary warmup, prefix-aware routing, speculation, quantization, or P/D disaggregation one at a time.
• Replay bursts, cache loss, deploys, scale-up, replica loss, and recovery.
• Add automatic rollback for quality, policy, error-rate, and p95/p99 latency regressions.

Exit condition: each production route has a measured Pareto envelope and a tested rollback.

The road to real-time agents

The next frontier is not simply faster matrix multiplication. It is coordination between application semantics and serving state.

1. SLO-aware reasoning. Allocate thinking based on task difficulty, risk, and remaining time budget.
2. Session-aware cache policy. Use stable, privacy-preserving session and intent hints for routing, prefetch, eviction, and offload.
3. Dynamic speculation. Adapt draft length and verification strategy to domain, concurrency, and acceptance behavior.
4. Elastic prefill/decode capacity. Scale phases independently without making KV transfer the new bottleneck.
5. End-to-end provider routing. Choose accelerator and service tier using accepted-outcome latency, not model marketing alone.
6. Predictive warmup. Compile shapes, load weights, and pre-position cache state before a known traffic transition.
7. Latency-aware prompt optimization. Search prompt quality, prefix reuse, output length, and cache-write economics together.
8. Portable latency traces. Compare hosted and self-hosted routes with the same stage vocabulary and scorecard.

vLLM’s public Q3 2026 roadmap already points toward agent hints for prefix policy, multi-tier KV offload, tuned disaggregation, dynamic speculative decoding, lower cold starts, and more interactive tokens. SGLang’s session-aware radix cache is an early implementation of the same direction.

Real-time agents will emerge when the runtime knows enough about the task to allocate compute and state intelligently—without letting the model silently redefine the budget or authority. That is latency engineering: not making every token fast, but making the critical path deliberate.

What to read next

• Prompt Engineering and LLM Latency: The 10-Day Field Report
• Production Prompt Engineering in 2026: From Instructions to an Evaluated Contract
• AI Gateway and LLM Routing in Production
• Tokenomics: Cost per Trusted Outcome