Nemotron Nano 9B v2 · hybrid Mamba-2State snapshots 6.6× smaller than int8, at better quality.
A provider serving Nemotron on 1,000 H100s.
Each GPU keeps 200 conversation states cached so returning users, branches and rollbacks skip reprocessing their prompt. In bf16 that cache takes 14.2 GB of every card. With weights.zip it takes 1.1 GB, and the memory it frees serves more users.
Capacity · Nemotron Nano 9B v2, 100k contextCached states in the same 14.2 GB per GPU
Resume history for 100,000 users
32k conversations, a checkpoint every 256 tokensGPUs' worth of memory holding cached states
Annual GPU cost of the state cache
| State format | Memory per GPU | GPUs' worth | Per year, reserved | Saving, reserved | Saving at on-demand rates |
|---|---|---|---|---|---|
| bf16 | 14.2 GB | 177 | ~$3.1M | baseline | baseline |
| int8 | 7.3 GB | 91 | ~$1.6M | ~$1.5M | ~$2.7M |
| weights.zip | 1.1 GB | 14 | ~$0.25M | ~$2.9M | ~$5.2M |
Against int8, weights.zip saves ~$1.4M a year reserved and ~$2.4M on demand. GPUs' worth is the state cache's share of the fleet's 80 GB cards; memory freed from the cache serves more users, so the fleet needs that many fewer GPUs. Reserved H100 capacity at about $2.00 per hour ($17,500 per GPU-year); on-demand at about $3.61 per hour. "ours" is weights.zip.
Per conversationOne Nemotron Nano 9B v2 state snapshot is 70.8 MB in bf16. Checkpointing a 32k conversation every 256 tokens, as prefix caching and rollback need, adds up to 9.1 TB for every 1,000 users.
Size of one state snapshot
Nemotron Nano 9B v2, 100k contextSnapshot storage per 1,000 users
32k conversations, a checkpoint every 256 tokensBar widths are drawn to scale. "ours" is weights.zip.
What you get
- Keep many more conversation states cached, so returning users skip reprocessing their prompt.
- Cheaper conversation resume, branching and speculative rollback.
- Output quality better than int8 at 100k context, where int4 is unusable.
- No retraining and no changes to the model.
Good fit when
- You serve Nemotron or other hybrid Mamba models.
- You use prefix caching or keep conversation state between turns.
- Snapshot storage or cache hit rate limits your throughput.
Measured on live model output against the uncompressed model.
State snapshot formats · Nemotron Nano 9B v2| Format | Snapshot size | Quality against int8 | |
|---|---|---|---|
| bf16 (uncompressed) | 70.8 MB | reference model | |
| int8 (block 64) | 36.5 MB | baseline | |
| int4 (block 64) | 18.8 MB | 14 to 60× more error: unusable | |
| weights.zip, 100k context | 5.5 MB | lower error than int8 | |
| weights.zip, 100k context | 4.4 MB | about int8 level | |
| weights.zip, 16k context | 2.0 | 8.9 MB | equal to int8 |
Error is KL divergence of the model's next-token predictions against the uncompressed model. At 100k context, weights.zip's advantage over int8 is 6 to 12 standard errors. At 16k the saving over int8 is 4.1×.
Repeatable across runs, models and content- Three independent runs all beat int8 on snapshot size and quality at 100k context.
- The result holds on the smaller Nemotron-3 Nano 4B.
- It holds on Python source code as well as prose: int8 quality at 100k.
- Every number was re-measured by an independent verifier, with sizes counted from the real stored bytes and the test text kept separate from any tuning data.
Ships with the model. Nothing to retrain.
- Prepared once per modelA one-time offline step per model. Nothing is added to each snapshot, and the model weights are untouched.
- Store snapshots compressedSnapshots are written and read in the compressed form, in memory or on disk.
- Pilot on your trafficWe integrate with your serving stack and measure snapshot size and quality on your prompts.
Free benchmark on your models and traffic.
- Tell us your stackThe models you serve, your context lengths and your serving engine.
- Get your numbersMemory saved, quality against your current format and capacity per GPU, measured on your prompts.
- PilotRun on one workload for 30 days. Pricing is a share of the measured saving.
Built for the models you serve.
Model names and logos are trademarks of their owners and are used only to identify the models tested. weights.zip is not affiliated with or endorsed by DeepSeek, NVIDIA, Z.ai, Meta or MiniMax.