weights.zip KV ExactEarly access

A third smaller KV cache. Not one bit changed.

1.46 to 1.49× more conversations per H100, bit-identical output. KV Exact stores your KV cache in 0.687 of its original size. The cost is stated plainly: attention runs at about 0.6 to 0.8× speed per step on H100 (measured, 2026-10-10).

Request early access See the benchmarks
Tested on
DeepSeek-V2-Lite GLM-4.7-Flash Llama-3.1-8B class
Cache for 100 conversations of 32k tokens: original 102 GB, weights.zip 70 GB
Cache for 100 conversations of 32k tokens · DeepSeek-V2-Lite
1.46 to 1.49×more conversations per H100 Bit-identical output. Measured on H100, 2026-10-10, DeepSeek-style models.
0.687of original size 32% fewer bytes to store and move.
0.6 to 0.8×attention speed per step The cost, measured on H100: median 0.63, best later kernel median 0.77, range 0.55 to 1.03.
0quality risk Bit-identical output. Not one bit changed.
Inference · on the GPU
GPU impact

More conversations on the same H100.

Inference is the biggest line on the bill, and the KV cache caps how many conversations each GPU can serve. KV Exact frees about 30% of the memory the cache uses so each H100 holds 1.46 to 1.49× more conversations, with output identical bit for bit. The cost is speed: attention runs at about 0.6 to 0.8× speed per step on H100 (measured, 2026-10-10).

GPU memory for 100 conversations

32k tokens each, lower is better GPU memory for 100 conversations of 32k tokens: standard cache 102 GB, KV Exact 71 GB

Conversations per 80 GB H100

By conversation length, higher is better
Per conversation, DeepSeek-V2-LiteStandard cacheKV Exact
GPU memory, 32k tokens1.02 GB~0.71 GB
Share of one 80 GB H1001.3%0.9%
Conversations per H100 at 32k / 64k / 128k39 / 19 / 955 / 27 / 13
Answer qualityreferencebit-identical
The cost, stated plainly · attention speed against the uncompressed bf16 cache, same batch, DeepSeek-style cache, H100 (measured, 2026-10-10)

Attention speed per step on H100

1.00× is the uncompressed cache

Range across configurations 0.55 to 1.03. You trade some attention speed per step for 1.46 to 1.49× more conversations on the same GPU.

Conversations per H100

Same memory, higher is better

Measured on H100, 2026-10-10, bit-identical output.

A provider on 1,000 H100s · the same conversations, fewer GPUs (estimate)

GPUs for the same traffic

Annual GPU cost, reserved

~$5.0Mestimated saved a year at reserved rates, ~$9.0M at on-demand rates, for a fleet limited by memory, with the same answers bit for bit.
Fleet limited byGainH100s neededSaving, reservedSaving, on demand
GPU memory (concurrent conversations), estimate1.4×715~$5.0M~$9.0M
Standard-attention models (Llama and Nemotron style). Size is confirmed on a real Llama-3.1-8B-class cache: 0.686 to 0.687 of original, bit-exact. Speed on real standard-attention caches is still pending, so every capacity, speed and fleet figure above is for DeepSeek-style caches only.

DeepSeek-V2-Lite cache in bf16 is 31,104 bytes per token. Per-GPU figures assume about 40 GB of each H100 is left for cache after the model weights. KV Exact frees about 30% of cache memory including memory-page overhead (measured on one long prompt; more prompts pending) and fits 1.46 to 1.49× the concurrent long-context conversations at the same cache memory on an H100 (measured; first verified at about 1.4× on a consumer GPU). The 1.46 to 1.49× conversation capacity and the 0.6 to 0.8× attention speed were measured on an H100 on 2026-10-10 with bit-identical output; the 1.4× in the fleet estimate is the more conservative figure first verified on a consumer GPU. Reserved H100 capacity at about $2.00 per hour ($17,500 per GPU-year); on-demand at about $3.61 per hour.

Offload & transfer · off the GPU Performance improvements

A third less traffic. Faster restores on a consumer GPU.

Restoring a conversation from CPU memory or SSD, reusing a shared prompt prefix, and handing a request from a prefill server to a decode server all wait on a link. KV Exact moves a third fewer bytes over every one of them, and keeps answers bit-identical.

Restore speed

consumer GPU, higher is better

Bytes moved

lower is better

Offload speed

consumer GPU, higher is better

Cache per GB

RAM or SSD, higher is better
MetricOriginalweights.zipImprovement
Restore from CPU memory over PCIe (consumer GPU)1.00×1.43×43% faster
Bytes moved for offload, restore and prefill to decode1.00×0.68×32% less
Offload to CPU memory over PCIe (consumer GPU)1.00×1.35 to 1.43×up to 43% faster
Conversations cached per GB of RAM or SSD1.00×1.46×46% more

Restore and offload measured on a consumer GPU with a 113 MB conversation cache; data-center GPU timing is pending. In a fleet that spends 10% of GPU time waiting on cache restore or transfer, cutting that wait by 32% frees about 3% of GPU time: about $30M a year on a $1B inference budget. That is an illustration, not a measurement; your share depends on how link-bound your cache traffic is.

Benchmarks

A third smaller, more conversations per GPU, and bit for bit.

Total size (lower is better)

Restore speed, consumer GPU (higher is better)

How it was verified

  • Every stored byte is counted from the real file.
  • A separate decoder rebuilds the cache bit for bit.
  • Sizes are measured on held-out conversations never used for tuning.
  • Model output with the cache restored inside attention is bit-identical from 32k to 128k tokens.

Key takeaways

  • 0.687 of original size (32% fewer bytes).
  • 1.46 to 1.49× more conversations per H100, measured.
  • The cost: attention runs at about 0.6 to 0.8× speed per step on H100.
  • 1.43× faster restore (consumer GPU).
  • Bit-identical output. Not one bit changed.
DeepSeek-V2-LiteEvery test on this page: size, attention speed, restore, offload, GPU memory and conversations. GLM-4.7-Flash0.687 of original size, bit-exact, with no retuning. Same cache style as DeepSeek. Llama-3.1-8B class0.686 to 0.687 of original size on a real standard-attention cache. Speed testing next.
Full results · inference, on the GPU
MeasureResultStatus
Concurrent conversations per H100 at fixed cache memory, DeepSeek-style models1.46 to 1.49×measured on H100, 2026-10-10, bit-identical output
Attention speed per step against uncompressed bf16 at the same batch, DeepSeek-styleabout 0.6 to 0.8×measured on H100, 2026-10-10: median 0.63, best later kernel median 0.77, range 0.55 to 1.03
GPU memory freed for the KV cache, including page overheadabout 30%measured on one long prompt; more prompts pending
Model output with the cache restored inside attention, 32k to 128k tokensbit-identicalmeasured
Cache size on a real standard-attention model (Llama-3.1-8B class), keys and values0.686 to 0.687measured, 8k-token reference
Speed on standard-attention modelspendingnot yet tested on real standard-attention caches
Full results · offload, storage and transfer, off the GPU
MeasureResultStatus
Cache size, bit-exact, held-out data0.687 of originalindependently verified: real file written and rebuilt bit for bit by a separate decoder
Cache size, GPU version0.683 of originalindependently verified, bit-exact
Cache size on GLM-4.7-Flash, same cache style as DeepSeek, no retuning0.687 of originalmeasured, bit-exact
Conversation restore from CPU memory over PCIe1.43× fastermeasured on a consumer GPU (RTX 4060)
Conversation offload to CPU memory over PCIe1.35 to 1.43× fastermeasured on a consumer GPU (RTX 4060)
Cached conversations per GB of CPU RAM or SSD1.46×follows from size

As of October 2026, none of the major open serving stacks (vLLM, SGLang, TensorRT-LLM, LMCache) ships lossless KV compression. Serving Llama? KV Cache stores Llama caches 1.59× smaller, also bit for bit.

What you get

  • Zero quality risk. Output is identical, so there is no per-model accuracy sign-off, no A/B test and no regression risk.
  • 1.46 to 1.49× more conversations on every H100, on DeepSeek-style models, at a cost of about 0.6 to 0.8× attention speed per step.
  • 1.43× faster conversation restore and up to 1.43× faster offload over PCIe (consumer GPU).
  • A third less network traffic between prefill and decode servers, and 1.46× more cache in the RAM and SSD you already own.

Good fit when

  • You turned down FP8 or int4 KV caches over quality risk.
  • You run regulated or accuracy-critical products, or long reasoning chains where small errors add up.
  • GPU memory caps your concurrency, or you offload cache and run disaggregated prefill and decode.
Deployment

Nothing to sign off. Nothing changes but the bill.

  1. Drop in wherever the cache livesGPU memory during inference, CPU memory and SSD offload, prefix caches, and prefill to decode transfer. The model and its outputs are untouched.
  2. Checked bit for bitEvery restored cache matches the original exactly, so there is no accuracy test to run before going live.
  3. Pilot on your trafficWe measure concurrency, cache hit rate and the GPU time freed on your models and links. Standard-attention models such as Nemotron, FP8 caches and model weights are in testing now.
Start with your own data

Free benchmark on your files and hardware.

  1. Send a sampleA few models, a checkpoint series or a KV cache dump, or run our benchmark inside your account.
  2. Get your numbersBytes saved, restore speed on your GPUs, and a byte-for-byte match report.
  3. PilotRun on one workload for 30 days. Pricing is per terabyte managed or a share of the measured saving.
Request a pilot
The product line

Ten products for the data inside AI models.

How these numbers were measured

All figures are measured on public models (Qwen2.5 0.5B and 1.5B, Llama-3.1 8B and 70B, Pythia-160M, GPT-2 and a 27B FP8 model) on NVIDIA H100 SXM, H200 and RTX 4060 GPUs. Every restore was checked byte for byte against the original. zstd comparisons use zstd level 19 or NVIDIA nvCOMP Zstd on the same files. Dollar figures assume S3 Standard-class storage at about $24 per TB-month and typical cloud egress pricing. Results depend on the model and data type, so the free benchmark measures yours.