Adaptive or auditable: pick one
Every serious LLM serving stack eventually rediscovers the same fact: the KV cache is the real estate. Each token a model has read occupies cache memory until its request completes, and on production GPUs that memory, not compute, caps concurrency. A single long-context request holds the memory of dozens of short ones. The industry's response has been a family of methods that evict: discard cached entries mid-flight and reclaim the space.
But the question that decides whether eviction is production-grade is not how much memory it saves. It is this: who decided what to discard, and can you defend that decision afterward? Sort the published and deployed methods by how they answer, and the field collapses into two families.
The adaptive family
H2O keeps heavy hitters, tokens that accumulate high attention scores. SnapKV selects entries by attention patterns observed over a recent window. Dozens of descendants refine the recipe: better scoring, smarter pooling, per-head budgets. What every member shares is the decision procedure. Runtime attention statistics choose what lives.
This is genuinely adaptive. The keep-set follows the content of the request, which is why these methods post strong average-case numbers. But look at what the decision depends on: attention scores computed during this particular run, downstream of batch composition, kernel scheduling, and floating-point accumulation order. Two consequences follow, and neither is fixable by a better scoring function.
The decision cannot be predicted before the request runs, so capacity planning against it is guesswork, and no admission controller can know at enqueue time what a request will actually cost. And the decision cannot be meaningfully audited after the fact. Ask why an entry was discarded and the only true answer is that its score was low at that moment, in that batch, on that kernel. That is a description, not a justification, and in general it is not even reproducible.
The failure mode this buys is silent and input-dependent. Public benchmarking was slow to surface it because the standard long-context suites are prose-heavy: aggregates dominated by summarization and open-ended QA, where lossy compression looks benign. The purpose-built suites tell the other story. RULER's multi-key retrieval tasks and SCBench's string-retrieval tasks show compression and eviction methods degrading sharply on exactly the requests where a specific span of the input must survive verbatim, while the same methods' aggregate scores stay respectable. Which of tomorrow's production requests fall in the failing class? Unanswerable in principle: the method itself does not know until runtime.
An engineering organization can ship that. It cannot certify it, bound it, or write it into an SLA.
The auditable family
The opposite corner is occupied by positional schemes, of which StreamingLLM is the cleanest: keep the attention sinks at the start of the sequence, keep a sliding window at the end, discard the middle. The rule is fixed. It is fully deterministic, trivially predictable, reproducible everywhere, auditable in one sentence.
It is also content-blind. The keep-set is decided by position alone. The same entries survive whether they contain the answer to the user's question or boilerplate. For open-ended streaming chat that is a defensible trade. For traffic where the payload matters, retrieval, code, documents, anything where the answer lives in the middle, it certifiably discards without regard to relevance. You can audit the decision perfectly. The decision just never looked at your request.
The dichotomy
Line the two families up and the pattern is exact. Adaptivity has been purchased with runtime statistics, at the price of auditability. Auditability has been purchased with content-blindness, at the price of adaptivity. Every method in the literature sits at one end or the other, because the field has treated adaptive and statistical as synonyms.
Production serving needs both ends at once. SLAs, compliance regimes, and incident forensics demand decisions that are reproducible and explainable: worst-case properties. Real traffic is not positionally uniform: an average-case property. The current menu forces an operator to choose which one to abandon.
A method whose eviction decision adapts to the content of the request, but is a pure function of that request, with no runtime state, no scores, no learned parameters, would hold both ends simultaneously. The keep-set would be known at enqueue. An audit would reduce to re-running the decision. The behavior would be adaptive because the decision actually reads the request.
That is the bar we set for ourselves. We are not going to explain how we cleared it: US application 19/546,901 will publish in due course. But whether we cleared it is checkable on your own traffic in an afternoon, and the first arm of the check is a byte-for-byte diff. The protocol is on the front page.