Models · NVIDIA Nemotron 3.5 Super 120B-A12B

Up to $20M a year back on a 5‑trillion‑token week.

Hybrid Mamba state snapshots 6.6× smaller than int8 and 25.6× smaller than fp32, so a long conversation resumes from a stored state instead of being read again from scratch. The saving is an estimate built on measured sizes and quality. Pilots are open.

Request a pilot See the numbers

Two equal-height stacks of cache memory: six int8 state snapshots on the left, forty weights.zip snapshots on the right, 6.6 times as many.

$1M to $20Ma year in prefill GPU cost, estimate

6.6×smaller than int8, measured

25.6×smaller than fp32, measured

1.04 to 1.24×int8's error, held-out text and code

Estimated yearly prefill GPU cost saved at 5 trillion tokens a week and $9 per GPU-hour: $15.2M if cache hits rise 55 points and prefill runs at 20,000 tokens per second, $3.8M at 80,000 tokens per second, $1.0M for a 15 point gain at 80,000.

Estimate. Yearly prefill GPU cost at $9 per GPU-hour for an NVIDIA B300, for 221 trillion input tokens a year. The cache hit-rate gain is assumed, not measured.

The money

A 5-trillion-token week is a $53M a year business.

At open-model prices for a 120B-A12B model, $0.15 per million input tokens and $0.50 per million output tokens as listed on OpenRouter on 17 October 2026, five trillion tokens a week is about $53M of revenue a year. Input tokens are 85% of the volume, and every one of them has to be read by a GPU first.

	Tokens per year	Price per million	Revenue per year
Input	about 221T	$0.15	about $33M
Output	about 39T	$0.50	about $19.5M
Total	260T		about $53M

The whole prefill bill for those 221T input tokens is about $7M to $28M a year at $9 per GPU-hour, for prefill speeds of 80,000 down to 20,000 tokens per second. No saving can be larger than that bill, so the estimates below are shares of it: 15% to 55%.

Prefill saved, estimate

Four scenarios, none of them the best case.

Each scenario assumes the cache hit rate rises by the stated points, because a state 6.6× smaller lets many more long conversations stay cached. A cache hit skips reading the prompt again. At 168 MB per fp32 snapshot, a multi-turn agent workload often cannot keep enough snapshots, so most long contexts are recomputed. Making them small is what makes the cache work at all.

Scenario, input tokens not re-read	$9 per GPU-hour	$12 per GPU-hour
+55 points of hits (122T tokens a year), 80,000 tokens per second	about $3.8M	about $5.1M
+55 points of hits (122T tokens a year), 20,000 tokens per second	about $15M	about $20M
+15 points of hits (33T tokens a year), 80,000 tokens per second	about $1.0M	about $1.4M
+15 points of hits (33T tokens a year), 20,000 tokens per second	about $4.1M	about $5.5M
Revenue-equivalent of the tokens not re-read, at $0.15 per million	about $5.0M (+15 points) to $18.2M (+55 points)

So the honest range is about $1M to $20M a year of GPU cost at 5 trillion tokens a week. The hit-rate gain is the soft spot: it is assumed, and a replay of real multi-turn sessions would turn the 15 to 55 point range into one number. It is the most valuable next test.

Why it is measured in snapshots

One user's state: 25.6× smaller.

One user's full Mamba state on Nemotron 3.5 Super is 167.8 MB in fp32, 43.3 MB in int8 and 6.55 MB with weights.zip: 25.6 times smaller than fp32 and 6.6 times smaller than int8.

Measured on Nemotron 3.5 Super 120B-A12B at 100k context, on one B300. Sizes are counted from the real stored bytes.

Format	One user's state	weights.zip is smaller by
fp32	167.8 MB	25.6×
int8	43.3 MB	6.6×
weights.zip	6.55 MB

Quality, measured

Close to int8, 10 to 30× more accurate than int4.

On held-out test windows at 100k context, weights.zip has 1.04 to 1.24× the error of int8 while storing 6.6× less. Top-1 token agreement with the unchanged model is 92 to 100%. The Nano 9B reached 0.78× int8's error, so on Super the result sits slightly above int8, close to int8 quality.

Test window	int8 error	weights.zip	Against int8	int4 error
Text, offset 160k	0.00579	0.00599	1.04×	0.0569
Text, offset 130k	0.00550	0.00587	1.07×	not measured
Python code, never seen in tuning	0.00236	0.00292	1.24×	0.0845

Error is KL divergence of the model's next-token predictions against the unchanged model. Every test window was held out from tuning.

What moves the number

It scales with volume, price and response time.

Estimated yearly prefill GPU cost saved in the high case: $152M at 50 trillion tokens a week, $76M at 25 trillion, $15.2M at 5 trillion.

Volume

Every figure scales linearly. A provider serving 50 to 100 trillion tokens a week on hybrid models multiplies the estimates by 10 to 20.

Price

The revenue-equivalent rises by the same factor as the price per token. A model sold nearer frontier prices is worth 10 to 100× more per token than these open-model rates.

Response time

On this test setup a 100k-token cache hit restores in about 7 seconds, against 60 to 96 seconds of prefill. The setup is unoptimized and production would be faster for both. For agents, that is a product-quality gain on top of the cost.

Hybrid models carry only part of their context in the state snapshot, so the real share of traffic that benefits is unmeasured. A replay on real multi-turn sessions will settle it.

Speed, measured

Saved in 82 ms, restored in 4.9 ms.

Encoding a full Super state on the GPU now takes 82 ms, down from 787 ms on the CPU, which is 9.6× faster. The restore runs on the GPU in 4.89 ms and is bit-exact against the reference decode. The state format is unchanged, and the size and error are the same as before.

	Before (CPU)	Now (GPU)
Encode a full state	787 ms	82 ms
Restore a state	660 ms	4.89 ms, bit-exact

Measured on one B300 with real Nemotron 3.5 Super states. Reconstruction error is 1.000× the CPU result.

The whole session, measured

The KV cache shrinks too: 31% off a whole session.

Super keeps both a state and a KV cache per user. With our state compression and a compressed KV cache together, a whole session falls from 463 MB to 321 MB at the balanced setting, or 295 MB at the maximum setting. Error stays within 1.09× of the baseline on the text windows and 1.23× on the code window at the balanced setting.

Setting	Text, 160k	Text, 130k	Code, 200k	Session	Smaller by
Baseline: fp8 K, int8 V	0.0088	0.0215	0.0060	463 MB
weights.zip, balanced	0.0096 (1.09×)	0.0231 (1.08×)	0.0074 (1.23×)	321 MB	31%
weights.zip, maximum	0.0103 (1.17×)	0.0264 (1.23×)	0.0120 (2.0×)	295 MB	36%

Error is KL divergence against the unchanged model on held-out windows. Balanced is the best trade-off. The maximum setting is for workloads that accept more error on code.

Status

What is measured and what is not.

Sizes, error, encode and restore speed are measurements on a real model. The dollar figures on this page are estimates, and the cache hit-rate gain behind them is assumed until a replay of real sessions measures it.

The smaller Nemotron Nano 9B v2 results are on their own page.

Get the number for your traffic.

Free benchmark on your models and prompts. We measure the state size and quality on your traffic, then replay your sessions to turn the hit-rate range into one figure.

Request a pilot

NVIDIA, Nemotron and B300 are trademarks of NVIDIA Corporation. weights.zip is not affiliated with or endorsed by NVIDIA. OpenRouter prices change; the figures use the prices listed on 17 October 2026.

Start with your own models

Free benchmark on your models and traffic.

Tell us your stackThe models you serve, your context lengths and your serving engine.
Get your numbersMemory saved, quality against your current format and capacity per GPU, measured on your prompts.
PilotRun on one workload for 30 days. Pricing is a share of the measured saving.

Request a pilot

Models

Built for the models you serve.

Model names and logos are trademarks of their owners and are used only to identify the models tested. weights.zip is not affiliated with or endorsed by DeepSeek, NVIDIA, Z.ai, Meta or MiniMax.