Up to 2.5× the long-context users per GPU. Faster than fp8.

A provider running 1,000 H100s for long-context serving.

Today that fleet holds its KV cache in fp8. Serving Llama-3.1-8B it handles 6,000 concurrent 128k-context users; serving Llama-3.1-70B, 2,375. With weights.zip it serves the same users on 40 to 50% of the GPUs, or up to 2.5 times the users on the same fleet.

Reserved H100 capacity at about $2.00 per hour ($17,500 per GPU-year); on-demand at about $3.61 per hour. Above 32k context the KV cache dominates inference cost, so GPU memory sets how many users each card can serve. For a lab spending $1B a year on inference, the measured saving on 8B-class models is about $8M to $15M a year.

GPUs are counted in whole eight-GPU nodes: 19 users per node with fp8, 34 with int4, 38 with 4-bit and 44 with 3.5-bit. Same reserved and on-demand H100 rates as above.

Pricing is a share of the measured saving, typically 15 to 20%: a site license for the largest fleets, a metered platform license for API clouds, and a commercial SDK license for smaller teams.

Users per GPU are concurrent 128k-token sessions of Llama-3.1-8B on one 80 GB H100. On the same card fp8 fits 6 and int4 fits 12. For Llama-3.1-70B on an 8x H100 node, fp8 fits 19 and int4 fits 34.

Measured on NVIDIA H100 with Llama-3.1-8B and 70B, out to 128k tokens.

The KV budget is 92% of GPU memory less 16.06 GB of bf16 weights and 4 GB of activations: 58.6 GB. Per-model overhead is about 1.5 MiB and is included.

141 GB of bf16 weights across the node, with the same memory rules as above, leaves 416 GB for the KV cache.

fp8 is NVIDIA's TensorRT-LLM XQA kernel through FlashInfer 0.7.0, running at about 2.8 TB/s. int4 is a kernel running at about 2.9 TB/s, close to the H100's memory bandwidth limit. At 32k and 64k context with batch 8 to 64, the 4-bit format runs 1.31 to 1.53× faster than fp8 and the 3.5-bit format 1.57 to 1.87×. At batch 8 and 128k, the 3.5-bit format cuts decode attention time by 42% against fp8. Every configuration decodes bit for bit identically to its reference decoder.

The per-GPU shape of a 70B model served across 8 GPUs, against NVIDIA's TensorRT-LLM XQA kernel for fp8. These are the same configurations as the 70B quality results below. The ratios hold when the 70B is served on 4 GPUs or on 1.

Scored at every position of two held-out 131,072-token windows against the bf16 model, with no subsampling. Error is average token KL divergence. Words changed is the share of positions where the most likely next token differs from bf16.

44 RULER-style prompts (22 per length), every format answering the same ones, H100. The 3.5-bit format sits below int4 at these lengths but within measurement noise, at 22% less memory. Pooled over 64k, 96k and 128k (66 prompts) it scores 89.4% against int4's 90.9% and bf16's 93.9%.

Two held-out 131,072-token texts, every position scored against bf16, with no test text used in calibration. On text A the 4-bit format agrees with the uncompressed model's top token 93.8% of the time, against int4's 93.7%. No late-context error spike out to 128k tokens.

72 prompts, every format answering the same ones. The 4-bit format is within measurement noise of int4 (2.8 points, 95% interval 8.2 below to 2.7 above) and gave the same answers as int4 on every 96k prompt. These prompts are hard for the 70B even uncompressed, so most sit near the edge of what it can retrieve.

No retraining. No changes to the model.

1. Calibrate onceA one-time offline step per model on a single GPU, using two long text windows. Llama-3.1-70B calibrates in about 15 minutes on one H100. No training, and the weights are untouched.
2. Swap the kernelsDrop-in replacements for the decode attention kernel and the KV cache write path, working with a paged cache. Kernel settings are chosen automatically for each batch size and context length.
3. Pilot on your trafficWe integrate with your serving stack and measure capacity, speed and quality on your models and prompts.

Free benchmark on your files and hardware.

1. Send a sampleA few models, a checkpoint series or a KV cache dump, or run our benchmark inside your account.
2. Get your numbersBytes saved, restore speed on your GPUs, and a byte-for-byte match report.
3. PilotRun on one workload for 30 days. Pricing is per terabyte managed or a share of the measured saving.

Ten products for the data inside AI models.

All figures are measured on public models (Qwen2.5 0.5B and 1.5B, Llama-3.1 8B and 70B, Pythia-160M, GPT-2 and a 27B FP8 model) on NVIDIA H100 SXM, H200 and RTX 4060 GPUs. Every restore was checked byte for byte against the original. zstd comparisons use zstd level 19 or NVIDIA nvCOMP Zstd on the same files. Dollar figures assume S3 Standard-class storage at about $24 per TB-month and typical cloud egress pricing. Results depend on the model and data type, so the free benchmark measures yours.