One stack for every layer of AI serving and training

Running a large model costs money in more places than most dashboards show. There is the storage bill for every checkpoint and fine-tune. There is the egress and the wait each time a new GPU node pulls a model. There is GPU memory, which decides how many models and how many conversations fit on a card. And there is the time spent reading a long prompt before the first word comes back.

Most tools fix one of these. weights.zip is built to work on all of them, so a team can start where it hurts most and add layers as it grows.

What the stack covers

Think of it as following one model through its life. It is trained and saved, stored in a registry, loaded onto GPUs, held in memory, and then asked questions. Each of those steps has a weights.zip layer.

Layer	What changes for you	Result	Measured on
Storage	Model registries and fine-tune fleets take less space	up to 47% less space	fine-tune with its base model; 33% bf16 alone, 20% FP8
Checkpoints	Smaller checkpoints, faster recovery after a failure	44% smaller	restores at up to 21 GB/s per H100 (bf16)
Loading	New nodes start serving sooner	21% less egress	per FP8 model load
GPU memory	More models held on each card, switched in well under a second	1.5x bf16 models per GPU	per card; 1.26x for FP8
GPU to GPU	Gradients and weights cross the network in fewer bytes	24.5 to 49% fewer bytes	B200
Prefill	Long prompts are read faster	1.4x faster	H100, 14B model, 128k tokens
KV Exact	The conversation cache in less memory, identical output	0.687x size	bit-identical output
Compact KV formats	Many more long-context users per GPU	up to 2.7x users vs fp8	128k-token contexts
Tiered KV cache	More cached context per GB of memory and SSD	1.6x capacity	identical output

Every number above was measured, on the hardware and models listed beside it.

The bottom of the stack: bytes at rest and in motion

The first layers are about files. A model registry with a few base models and hundreds of fine-tunes is mostly the same kind of data stored again and again. weights.zip Store keeps a fine-tune up to 47% smaller beside its base model, a bf16 model 33% smaller and an FP8 model 20% smaller, inside your own cloud account, and every file comes back byte for byte.

Training teams feel the same pressure from checkpoints. They are large, they are written often, and when a node fails the whole job waits for one to load. Checkpoint makes the weights in a run of checkpoints 44% smaller, and restores them at up to 21 GB/s per H100 for bf16. Writes finish sooner and a restart costs less idle time.

Load handles the moment a new GPU node comes up. The model arrives with 21% less egress, and over the network the node starts serving about 20% sooner. From host memory, an H100 loads it 1.29x faster than a raw copy. When traffic spikes, that is the difference between scaling in time and dropping requests.

Between GPUs, training traffic shrinks too. Gradient exchange between nodes uses 24.5% fewer bytes for bf16 and up to 49% fewer for fp32, measured on B200 while using under 7% of one GPU. Every tensor comes back bit for bit.

The middle: GPU memory

Memory is where serving costs are decided. A card that holds one model can only answer for that model. Resident keeps models compressed in GPU memory, so each card holds 1.5x as many bf16 models (1.26x for FP8) and switches between them in well under a second. For teams serving many fine-tunes or many customers, that means fewer idle GPUs waiting for the one model they hold to be asked a question.

The top: answering long prompts

The last layers are about the request itself. A user pastes a 128k-token contract and asks a question. Two things happen.

First the model reads the whole prompt. Prefill makes that 1.4x faster than dense FlashAttention-3 on an H100, with a 14B model at full accuracy: 20 of 20 retrieval checks passed. Reading the prompt takes 29% less time, so the wait for the first word shrinks with it, and the gain grows with prompt length.

Then the model keeps what it read in the KV cache while it writes the answer. That cache is what limits how many long conversations a GPU can hold. KV Exact stores it at 0.687 of its size with output that is identical bit for bit. Measured on an H100, that gave 1.46 to 1.49x more conversations at once. Attention runs at about 0.6 to 0.8x speed per step, so the gain is capacity, not speed.

When a small loss of precision is acceptable, the compact 3-bit and 3.5 to 4-bit formats go further: up to 2.7x more users than an fp8 cache at 128k tokens. The tiered cache adds 1.6x capacity across GPU memory and SSD, with identical output.

Each layer stands on its own, so you can adopt one without the rest. Together they cover the whole path a model takes, from disk to answer.

How to read the numbers

Each result was measured separately, on its own hardware and models. They save different costs, so they do not multiply into one figure. The KV cache options are alternatives rather than a combination: KV Exact keeps output bit-identical, while the compact formats trade a little precision for more users. Prefill was compared with dense FlashAttention-3, not with a full serving stack.

We would rather show you a smaller number we measured than a larger one we did not.

Where to start

Most teams have one cost that clearly dominates. If storage bills or checkpoint time are the pain, start at the bottom of the stack. If GPUs sit idle holding one model each, start with Resident. If long prompts are slow or long conversations run out of memory, start with Prefill and KV Exact.

See every layer together on the Stack page, or request a pilot. We run the matching layer on your models and your GPUs and report the result against what you run today.