weights.zip 3-bit

1.5× the long-context users of int4. 2.7× fp8.

A 3-bit KV cache that keeps long-context retrieval at int4 level. Each 128k-token session takes a third less memory than int4 and five-eighths less than fp8, so every GPU serves far more long-context users.

For agents, document retrieval and code assistants serving long contexts Pilots open
Two equal-height stacks of GPU memory: six thick fp8 sessions on the left, eighteen thin 3-bit sessions on the right.
3.0bits per cached value, against 8 for fp8 and 4.5 for int4
18128k-context users of Llama-3.1-8B per H100, against 6 with fp8
51128k-context users of Llama-3.1-70B per 8x H100 node, against 19 with fp8
$11.7Msaved a year on a 1,000-GPU long-context fleet
What it saves

A provider running 1,000 H100s for long-context serving.

Today that fleet holds its KV cache in fp8. Serving Llama-3.1-8B it handles 6,000 concurrent 128k-context users; serving Llama-3.1-70B, 2,375. With weights.zip 3-bit it serves the same users on 33 to 38% of the GPUs, or 2.7 to 3 times the users on the same fleet.

Llama-3.1-8B · one model copy per H100, 6,000 users with fp8
Annual GPU cost for 6,000 concurrent 128k-context users of Llama-3.1-8B
KV cache formatH100 GPUsPer year, reservedSaving vs fp8Saving at on-demand rates
fp81,000~$17.5Mbaselinebaseline
int4 (group size 32)500~$8.8M~$8.8M~$15.8M
weights.zip 3-bit334~$5.8M~$11.7M~$21.1M

Reserved H100 capacity at about $2.00 per hour ($17,500 per GPU-year); on-demand at about $3.61 per hour. Above 32k context the KV cache dominates inference cost, so GPU memory sets how many users each card can serve.

Llama-3.1-70B · the same 1,000 H100s as 125 eight-GPU nodes, 2,375 users with fp8
Annual GPU cost for 2,375 concurrent 128k-context users of Llama-3.1-70B
KV cache formatH100 GPUsPer year, reservedSaving vs fp8Saving at on-demand rates
fp81,000~$17.5Mbaselinebaseline
int4 (group size 32)560~$9.8M~$7.7M~$13.9M
weights.zip 3-bit376~$6.6M~$10.9M~$19.7M

GPUs are counted in whole eight-GPU nodes: 19 users per node with fp8, 34 with int4 and 51 with 3-bit. Same reserved and on-demand H100 rates as above.

Capacity

More sessions on every card.

Concurrent 128k-token sessions that fit after the model weights and activations. Bars are drawn to scale.

Memory per 128k-token session
FormatBits per valueLlama-3.1-8BLlama-3.1-70BCapacity vs fp8Capacity vs int4
bf161617.2 GB42.9 GB0.5×0.28×
fp888.6 GB21.5 GB1×0.56×
int4 (group size 32)4.54.8 GB12.1 GB1.8×1×
weights.zip 3-bit3.03.2 GB8.1 GB2.7×1.5×

The KV budget is 92% of GPU memory less the bf16 weights (16.1 GB for the 8B, 141 GB for the 70B) and 4 GB of activations per GPU. int4 stores one scale per 32 values, 4.5 bits per value overall.

Long-context retrieval

Finds what is in the context as well as int4 does.

Multi-key needle-in-a-haystack retrieval on Llama-3.1-8B at 96k and 128k tokens. Every format answers the same prompts, and the differences are paired. Multi-key retrieval at 96k to 128k: 3-bit scores 83.0%, within noise of int4 (85.2%); uncompressed bf16 scores 89.8% (H100, 2026-10-05, n=44 per length).

Format96k128kBoth
bf16 (uncompressed)95.5%84.1%89.8%
int4 (group size 32)88.6%81.8%85.2%
weights.zip 3-bit90.9%75.0%83.0%

44 prompts at each length, RULER-style, independently re-analysed from the raw outputs. Across both lengths the 3-bit format is 2.3 points from int4, with a 95% interval of 8.0 below to 3.4 above: within measurement noise, at a third less memory. At 96k it scored above int4.

What you get

  • 1.5× int4's long-context capacity and 2.7× fp8's, on the GPUs you already run.
  • Retrieval from long contexts at int4 level.
  • Error stays flat out to 128k tokens, with no growth late in the context.
  • No retraining and no changes to the model weights.

Best fit

  • Agents with long tool and conversation histories.
  • Retrieval over large document sets.
  • Repository-scale code assistants.
  • For the highest plain-text fidelity, the same family offers 3.5-bit and 4-bit formats.
Deployment

No retraining. No changes to the model.

  1. Calibrate onceA one-time offline step per model using two long text windows. No training, and the weights are untouched.
  2. Swap the kernelsDrop-in replacements for the decode attention kernel and the KV cache write path, working with a paged cache.
  3. Pilot on your trafficWe integrate with your serving stack and measure capacity, speed and retrieval on your models and prompts.
Start with your own data

Free benchmark on your files and hardware.

  1. Send a sampleA few models, a checkpoint series or a KV cache dump, or run our benchmark inside your account.
  2. Get your numbersBytes saved, restore speed on your GPUs, and a byte-for-byte match report.
  3. PilotRun on one workload for 30 days. Pricing is per terabyte managed or a share of the measured saving.
Request a pilot
The product line

Ten products for the data inside AI models.

How these numbers were measured

All figures are measured on public models (Qwen2.5 0.5B and 1.5B, Llama-3.1 8B and 70B, Pythia-160M, GPT-2 and a 27B FP8 model) on NVIDIA H100 SXM, H200 and RTX 4060 GPUs. Every restore was checked byte for byte against the original. zstd comparisons use zstd level 19 or NVIDIA nvCOMP Zstd on the same files. Dollar figures assume S3 Standard-class storage at about $24 per TB-month and typical cloud egress pricing. Results depend on the model and data type, so the free benchmark measures yours.