26 · Sweep Prompt & Output Length

Source: fanout.sh / inference-eng / 05-04 · ~9 min

NOTE

A request's shape (prompt length, output length) decides every number, so two teams can get very different tokens/s from the same model and GPU. Prompt length drives compute-bound prefill (TTFT), grows the KV cache, and slows decode for everyone sharing the batch. Output length drives the bandwidth-bound decode loop and how long memory is held. One output token costs about as much time as ~150 prompt tokens; price lists charge 5×. The same H100 fits 225 chat users but only 50 summarization users. Sweep one axis at a time, force the output length, and report every shape separately: a server has a map of tokens/s, not one number.

1. Two lengths, two machines

The prompt goes through prefill: all its tokens in one parallel pass, limited by compute, deciding time to first token. The output comes from decode: one token per step, each step reading the whole model, limited by memory bandwidth, deciding how long the rest of the answer takes.

A prompt token: 2 × 8B = 16 GFLOP at half of ~990 TFLOPS ≈ 32 µs (a 1,000-token prompt ≈ 32 ms). An output token with one user: 16 GB ÷ 3.35 TB/s ≈ 4.8 ms. Databricks measured the same: adding 512 prompt tokens raised latency less than adding 8 output tokens. Anthropic charges 5× more per output token on every current Claude model.

2. Sweep the prompt (output fixed at 256)

Prompt lengths 128, 512, 2k, 8k, 32k:

TTFT first grows in a straight line (prefill work ∝ tokens), then bends up: attention compares every token with every earlier one, which grows with length². Attention adds ~13% at 8k, ~54% at 32k, and at 128k costs twice everything else combined.

The prompt changes decode too. Every decode step reads the cache of every earlier token:

128 KiB per token for this model: a 1k context is 128 MiB, an 8k context a whole GiB, reread on every step. Long prompts slow TPOT too, for everyone sharing the batch.

3. Sweep the output (prompt fixed at 1,000)

TTFT barely moves; prefill didn't change. E2E = TTFT + TPOT × (N − 1) is a straight line with slope TPOT: at 12 ms per token, 256 tokens ≈ 3 s and 8,192 ≈ 98 s.

• That's where reasoning models live: their thinking tokens are output tokens, and each pays the decode price.
• Long outputs hold memory longer: a request decoding for 90 s keeps its cache for 90 s, so requests finished per second drops even when output tokens/s looks healthy.

4. Shape decides how many fit

After weights and overheads, the H100 has about 450,000 tokens of cache.

Chat (1k in, 1k out) ends at 2,000 tokens: ~225 fit. Summarization (8k in, 1k out) ends at 9,000: ~50 fit. Reasoning (1k in, 8k out) also ends at 9,000 but starts small, so the scheduler happily admits many, and later their caches grow into each other: preemption.

5. Running the sweep

vllm bench serve \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.1 \
  --ignore-eos \
  --max-concurrency 32

• --random-range-ratio makes each request vary a little instead of all being identical. SemiAnalysis's public benchmarks draw each prompt between 80% and 100% of the target.
• --ignore-eos is the flag people forget: without it the model stops when it decides it's done, and "8,000 output tokens" quietly becomes ~300.
• Change one axis at a time, in powers of two, and repeat each shape at a few concurrency levels.
• Record per point: TTFT and TPOT (median and tail), input and output tokens/s, cache usage, preemptions.
• Don't average different shapes into one number: the long requests hide inside the mean.

6. The shape map

Prompt on one axis, output on the other; each cell holds what the server does at that shape and whether it meets the latency target. Public benchmarks (InferenceMAX) pick three cells: 1k/1k chat, 1k/8k reasoning, 8k/1k summarization. Then pull real lengths from your logs and tune for where your traffic sits: a config that wins at chat can lose at summarization.

CAUTION

Three beliefs to drop. "Prompt tokens are free": cheaper, but they cost TTFT, compute and cache. "A server has one tokens/s number": it has a map. "max_tokens is the output length": it's only a ceiling unless you force the model to keep going.

7. Recap