Every layer of the stack, smaller and faster.
From the bytes on disk to the tokens streaming back, weights.zip works on each part of running a transformer: storage, loading, GPU memory, the links between GPUs, reading the prompt, the KV cache and generating the answer.
One long prompt, start to finish.
A user sends a 128k-token document and asks a question. Here is where the time and memory go, and what changes at each step.
-
Read the prompt
1.4x faster
Prefill reads the whole document 1.4x faster than dense FlashAttention-3, at full accuracy: 20 of 20 retrieval checks passed. Users wait less for the first word.
measured · H100 · 14B model, 128k tokens Prefill → -
Keep what it read
0.687x the memory
What the model learned from the prompt goes into the KV cache. KV Exact stores it at 0.687 of its size, and every bit comes back exactly.
measured · bit-identical output KV Exact → -
Stream the answer
1.46 to 1.49x more conversations
With a smaller cache, each H100 holds 1.46 to 1.49x more long conversations at once, with bit-identical output. Attention runs at about 0.6 to 0.8x speed per step, so the gain is capacity, not speed.
measured · H100 · bit-identical output KV Exact →
What we handle, layer by layer.
| Layer | What changes | Result | Measured on |
|---|---|---|---|
| Decode | More long conversations on each GPU, identical output | 1.46 to 1.49x more conversations | H100, bit-identical output |
| KV cache, bit-exact | The cache in less memory, identical output | 0.687x size | bit-identical output |
| KV cache, compact formats | 3-bit and 3.5 to 4-bit caches for long-context users | up to 2.7x users vs fp8 | 128k-token contexts |
| KV cache, tiered | More cached context per GB of memory and SSD | 1.6x capacity | identical output |
| Prefill | Long prompts read faster | 1.4x faster | H100, 14B, 128k tokens |
| GPU to GPU | Gradients and weights cross the network in fewer bytes, bit-exact | 24.5 to 49% fewer bytes | B200, 10 of 148 SMs |
| GPU memory | Models held compressed, switched in well under a second | 1.5x bf16 models per GPU | per card; 1.26x for FP8 models |
| Loading | New GPU nodes start serving sooner | 21% less egress | per FP8 model load |
| Checkpoints | Smaller checkpoints, faster recovery | 44% smaller | restores at up to 21 GB/s per H100 (bf16) |
| Storage | Model registries and fine-tune fleets | up to 47% less space | fine-tune with its base; 33% bf16 alone, 20% FP8 |
The fleet behind every answer.
GPU to GPU
Gradient traffic between nodes shrinks 24.5% for bf16 and up to 49% for fp32, at hundreds of GB/s on under 7% of one GPU. Every tensor comes back bit for bit.
measured · B200Checkpoints
Weights in a run of checkpoints are 44% smaller, so writes finish sooner and a restart after a node failure costs less idle time. →
measuredCold starts
When a new node comes up, the model arrives with 21% less egress and, over the network, starts serving about 20% sooner (calculated from the measured size). From host memory an H100 loads it 1.29x faster than a raw copy. →
measuredModel storage
Registries of base models and fine-tunes take up to 47% less space, inside your own cloud account: 33% for a bf16 model on its own, 20% for an FP8 model. →
measuredEach result was measured on its own, on the hardware and models listed with it. They apply to different costs, so they do not multiply into one figure. The KV cache options are alternatives: KV Exact keeps output bit-identical, while the 3-bit and 3.5 to 4-bit formats trade a little precision for more users. Prefill is compared with dense FlashAttention-3, not with a serving stack such as vLLM or TensorRT-LLM; neither was run.
Start with your bottleneck.
Tell us where your fleet spends its time and memory. We run the matching layer on your models and your GPUs and report the result against what you run today.
NVIDIA, H100, B200 and the NVIDIA logo are trademarks of NVIDIA Corporation. weights.zip is not affiliated with or endorsed by NVIDIA.