weights.zip Checkpoint

Smaller checkpoints. Faster recovery.

Smaller training checkpoints and faster recovery after a node failure. Your cluster spends less time idle while a checkpoint loads.

For labs and GPU clouds running large training jobs Pilots open

Training racks save thin green checkpoint plates into a tower half the height of a dashed outline showing raw size; a restore path flows back to the racks.

44%smaller weights, in a run of successive checkpoints (Pythia, step 140k to 141k)

up to 21 GB/srestore speed per H100 (bf16, current release)

up to 43 GB/srestore speed per H100 (FP8, current release)

On a $1 billion training budget, smaller checkpoints save an estimated $13.2M a year of GPU time spent waiting on checkpoints and $8.8M of checkpoint storage, $22M in total.

By what is checkpointed

What is checkpointed	Bytes vs raw
bf16 weights, single checkpoint	0.66
fp32 weights, single checkpoint	0.76
Weights, next checkpoint in the same run	0.56
Full checkpoint with optimizer state (16 bytes per parameter)	0.74

What you get

A quarter less checkpoint storage, with no change to what training sees.
Restores read fewer bytes from storage and unpack on the GPU faster than most storage can feed it.
Keep more checkpoints in the same budget, so you can roll back further.
Training resumes from exactly the state it saved. Nothing is rounded.

Good fit when

Checkpoints run to tens of terabytes or more.
Restores come from network or object storage.
Idle GPU time during recovery is a measurable cost.

Benchmarks

Fewer bytes written, and restores that outrun storage.

Checkpoint streams · size vs original, lower is better

What is checkpointed	Measured on	Size vs original
Next checkpoint, 1,000 steps after the last	Pythia-160M, fp32	0.562
Next checkpoint, 3,000 steps after the last	Pythia-160M, fp32	0.598
Full checkpoint, weights plus AdamW state	transformer training run	0.739
AdamW variance, the largest optimizer stream	transformer training run	0.660

Against zstd on the same weights

Model	Format	zstd	weights.zip	Less storage than zstd
Qwen2.5-1.5B	bf16	0.785	0.658	16%
Qwen2.5-0.5B	bf16	0.774	0.659	15%

Restore speed on the GPU

Decoder	Hardware	GB/s
weights.zip, FP8	NVIDIA H100 SXM	43.3
weights.zip, bf16	NVIDIA H100 SXM	21.2
weights.zip, bf16	RTX 4060	11.4
zstd on the GPU (nvCOMP), bf16	RTX 4060	2.1

Speeds are single-tensor figures for the current release. Every checkpoint was restored and compared bit for bit with the original.

Integration

Fits around the checkpoints you already write.

Compress checkpoint files once they are written, from the command line or from Python in your training job, and expand them on the GPU before training resumes. Your training code and its saved state do not change.

Supported

Weights in bf16, fp16, fp32 and FP8.
Optimizer state, including AdamW moments.
Checkpoint series from a single run, each stored against the one before.
Per-tensor access, so a restore can start before the whole file arrives.
Local disk, network file systems and object storage.

Tooling

# command line
weightzip compress    model.safetensors  model.wzst
weightzip decompress  model.wzst         model.safetensors
weightzip verify      model.safetensors  model.wzst

# Python
from weightzip import compress_file, decompress_file, verify_file
compress_file("model.safetensors", "model.wzst")
assert verify_file("model.safetensors", "model.wzst")

Verify it yourself

Don't take our word for it. Check every tensor.

Run verifyweightzip verify restores the file and compares every tensor with the original, bit for bit, and reports any difference.
Check with your own codeLoad both files with the standard safetensors library and compare the raw bytes of every tensor. The script below does it in a few lines.
Benchmark your filesSend us a sample or run our benchmark in your own account. You get bytes saved, decode speed on your GPUs and a byte for byte match report.

import torch
from safetensors.torch import load_file

a = load_file("model.safetensors")        # the original
b = load_file("restored.safetensors")     # after weightzip decompress
assert a.keys() == b.keys()
for name in a:
    x, y = a[name], b[name]
    assert x.dtype == y.dtype and x.shape == y.shape
    assert torch.equal(x.reshape(-1).view(torch.uint8), y.reshape(-1).view(torch.uint8))
print("identical:", len(a), "tensors")

Start with your own data

Free benchmark on your files and hardware.

Send a sampleA few models, a checkpoint series or a KV cache dump, or run our benchmark inside your account.
Get your numbersBytes saved, restore speed on your GPUs, and a byte-for-byte match report.
PilotRun on one workload for 30 days. Pricing is per terabyte managed or a share of the measured saving.

Request a pilot

The product line

Ten products for the data inside AI models.

How these numbers were measured

All figures are measured on public models (Qwen2.5 0.5B and 1.5B, Llama-3.1 8B and 70B, Pythia-160M, GPT-2 and a 27B FP8 model) on NVIDIA H100 SXM, H200 and RTX 4060 GPUs. Every restore was checked byte for byte against the original. zstd comparisons use zstd level 19 or NVIDIA nvCOMP Zstd on the same files. Dollar figures assume S3 Standard-class storage at about $24 per TB-month and typical cloud egress pricing. Results depend on the model and data type, so the free benchmark measures yours.