Read long prompts 40% faster.
Before a model writes a word, it has to read the whole prompt. weights.zip Prefill makes that step faster. On an H100, a 14B model reads a 128k-token prompt 1.4x faster than with dense FlashAttention-3, the fastest standard attention kernel on the chip, at full accuracy: 20 of 20 retrieval checks passed. Longer prompts gain more.
Same model, same prompt, same GPU. The timeline is scaled to the measured result: weights.zip finishes in 71% of the time.
The longer the prompt, the bigger the win.
Larger models gain a little less. Longer prompts gain a lot more. The 14B, 128k result is measured on H100; the other three are estimates from how the costs grow with model size and prompt length, not measurements.
Measured Estimated Ceiling as today's overheads shrink Dense FlashAttention-3 (1x)
| Model · prompt | Attention share of prompt time | Speed-up today | Ceiling |
|---|---|---|---|
| 14B · 128k tokens measured | 59% | 1.4x | 1.55x |
| 70B · 128k tokens estimate | about 47% | about 1.3x | about 1.4x |
| 14B · 1M tokens estimate | about 92% | about 1.8x | about 2.2x |
| 70B · 1M tokens estimate | about 88% | about 1.75x | about 2x |
At 70B the MLP is larger relative to attention, so attention is a smaller share of the time and the gain is a little lower. At 1M tokens attention costs 64 times what it does at 128k while the rest costs 8 times, so attention dominates and the gain grows.
Attention grows with the square. The rest, in a line.
- Only attention gets faster. The rest of the model, the MLP and projections, reads every token no matter what.
- Attention cost grows with the square of prompt length. Everything else grows in a straight line. Go from 128k to 1M tokens and attention costs 64x more; the rest, 8x.
- So the speed-up follows attention's share of the time. The longer the prompt, the more of the time is attention, and the more there is to win.
Cost relative to a 128k-token prompt.
Attention, the part that gets faster MLP and projections
Attention's share of prompt time
On a 14B model with a 128k-token prompt, attention is 59% of the time, measured. At 1M tokens it rises to about 92%, so nearly the whole prompt step is the part that gets faster.
A 70B model spends more of its time in the MLP, so at the same length its share is lower: about 47% at 128k and about 88% at 1M.
Anywhere the prompt is long.
Faster time to first token
Users wait for the whole prompt to be read before anything streams back. At 128k tokens the prompt is read in 29% less time, so that wait shrinks with it.
measuredFewer GPU hours on long documents
Contracts, codebases, transcripts and research packs are read in full on every request. Reading them faster frees GPU time for more traffic.
measured at 128kMillion-token contexts
The gain grows with length: an estimated 1.8x at 1M tokens on a 14B model, with room to about 2.2x as today's overheads shrink.
estimatePairs with KV Exact
Prefill reads the prompt faster. KV Exact then keeps the cache it writes a third smaller, bit for bit, for more conversations per GPU. →
product| GPU | NVIDIA H100 |
| Model and prompt | 14B model, 128k-token prompt |
| Baseline | The same model running dense FlashAttention-3, in our own test setup |
| Result | 1.4x faster prefill |
| Accuracy | Full accuracy: 20 of 20 retrieval checks passed |
The comparison is against dense FlashAttention-3, not against a serving stack such as vLLM or TensorRT-LLM; neither was run. The 70B and 1M-token figures are estimates: the measured 14B, 128k speed-up applied to how attention and the rest of the model scale, not measurements.
Measure it on your prompts.
We run your model at your prompt lengths on your GPUs and report prefill time and time to first token against dense FlashAttention-3.
NVIDIA, H100 and the NVIDIA logo are trademarks of NVIDIA Corporation. weights.zip is not affiliated with or endorsed by NVIDIA.
Free benchmark on your files and hardware.
- Send a sampleA few models, a checkpoint series or a KV cache dump, or run our benchmark inside your account.
- Get your numbersBytes saved, restore speed on your GPUs, and a byte-for-byte match report.
- PilotRun on one workload for 30 days. Pricing is per terabyte managed or a share of the measured saving.
Ten products for the data inside AI models.
All figures are measured on public models (Qwen2.5 0.5B and 1.5B, Llama-3.1 8B and 70B, Pythia-160M, GPT-2 and a 27B FP8 model) on NVIDIA H100 SXM, H200 and RTX 4060 GPUs. Every restore was checked byte for byte against the original. zstd comparisons use zstd level 19 or NVIDIA nvCOMP Zstd on the same files. Dollar figures assume S3 Standard-class storage at about $24 per TB-month and typical cloud egress pricing. Results depend on the model and data type, so the free benchmark measures yours.