Serve more sessions on the GPUs you already have.
A multi-codec plugin for vLLM that carries our codecs into your serving engine. Every session comes out smaller, whether it is still generating or parked, so each GPU serves more users and more conversations stay cached.
Long-context and agent traffic is memory-bound. Every open conversation holds GPU memory, every paused agent waiting on a tool call keeps holding it, and every cache miss pays for a full prefill again. HyperCache turns that memory into capacity.
It is not just offload. The codecs are the product.
Moving paused sessions off the GPU is a known technique, and serving engines already do it. The plugin is how our codecs get installed. What sets it apart is what the codecs do to the data.
More live sessions, not just parked ones
The codecs compress the cache of conversations that are still generating. That raises live capacity: up to 2.5 to 2.7× on Llama-style models against fp8, 1.3 to 1.5× on hybrid models, and 1.48× in the lossless mode. Offload tools do not do this.
Smaller parked sessions
Offload tools move raw or fp8 data. We move lossless data at about 0.58 to 0.68 of bf16, and hybrid-model state snapshots up to 25× smaller. More sessions stay cached and restores are faster, so cache-hit rates go up and fewer prompts are re-read.
Hybrid models
Standard offload tools handle attention KV. Mamba and linear-attention state snapshots are a gap, and our codec fills it: 82 ms to encode and 4.9 ms to restore on a B300.
Calibrated for each model
Every model gets its own calibration, and its bundle ships with a quality report.
Capacity, speed and launch-day support.
More concurrent sessions per GPU
Up to 2.5 to 2.7× more long-context users per GPU than FP8 on standard-attention models at 128k context, with quality that stays close to FP8.
1.3 to 1.5× more sessions on hybrid models (Mamba and linear attention), where the cache is already small.
A lossless mode with bit-identical output: 31% less cache memory, about 1.4 to 1.5× more sessions, with zero change to model output.
Paused sessions cost almost nothing
Agents and chat users pause between turns. Paused sessions move off the GPU automatically and come back before the next request starts.
A 100k-token hybrid-model session restores in about 5 to 10 ms, compared with seconds to re-read the prompt.
Up to 25× smaller session snapshots on hybrid models, so far more conversations stay cached in host memory and on disk. Higher cache-hit rates mean fewer prefill GPU-hours, which matters most when 90% or more of your traffic is input tokens.
Faster long prompts
Up to 1.4× faster prefill at 128k context with retrieval accuracy unchanged (needle-in-a-haystack 20/20).
Faster cold starts and smaller fleets
Models load up to 1.3× faster with bit-exact weights, so scale-out is faster.
Up to 1.5× more models resident per card for multi-model and fine-tune fleets.
Launch-day support for new models
The HyperCache Calibrator profiles a new open model, tunes HyperCache to it, runs our quality checks at 100k+ context, and outputs a ready-to-serve bundle.
Same-day bundles for the model families we already support: standard attention, latent attention, and Mamba and linear-attention hybrids. That covers most major open-model releases. Every bundle ships with its own quality report: accuracy at long context compared with your current serving format.
Zero manual tuning
One flag in your serving engine. No per-model knobs, no changes to your API, your clients, or your routing.
15 to 30% on long-context traffic.
| Your fleet | GPU spend | Typical saving on long-context traffic | Annual saving |
|---|---|---|---|
| 100 H100s | about $1.75M a year | 15 to 30% | $0.26M to $0.53M |
| 1,000 H100s | about $17.5M a year | 15 to 30% | $2.6M to $5.3M |
| 5,000 GPUs | about $90M a year | 15 to 30% | $13M to $26M |
Assumes about $2 per GPU-hour. The saving depends on the model, the context length and your cache-hit rate. Long-context and agent-heavy traffic saves the most.
The savings pool across all providers.
| Model | Weekly tokens (OpenRouter, this week) | Estimated GPU savings pool |
|---|---|---|
| Step 5 Preview | 17.1T | $13M to $120M a year |
| GLM 5.3 Flash | 12.7T | $3.5M to $13M a year |
| Muse Spark Open, at 50B tokens a day | $0.7M to $1.6M a year |
Lower cost per token lets you price below the market and still keep margin, or keep your price and widen it.
Teams that run out of memory first.
Inference providers
Competing on price and throughput across dozens of hosts serving the same model.
Agent platforms
Long, paused, multi-turn sessions.
Enterprises
Serving 100k to 1M-token contexts on a fixed GPU budget.
What is proven and what is in development.
Proven: lossless cache compression (bit-identical on H100), compact long-context caches on H100 at 128k, sparse prefill 1.4× on H100, and hybrid-model session snapshots 6.6× smaller than int8 at equal quality.
Measured on B300 (Nemotron 3.5 Super 120B, 100k context): sessions 31% smaller than FP8 with near-equal quality, snapshot restore 4.9 ms, encode 82 ms.
In development: the unified one-flag plugin and the launch-day Calibrator.
Now accepting design partners for pilots on your models and your GPUs, with savings measured on your own traffic.
Talk to us about a pilot.
Free benchmark on your models and traffic. Savings measured on your own prompts.
NVIDIA, H100 and B300 are trademarks of NVIDIA Corporation. weights.zip is not affiliated with or endorsed by NVIDIA. Dollar figures are estimates.