Fixing Gemma 4's oversized KV cache
Ardana v0.3.2 fixes an oversized Gemma 4 cache allocation. Model weights alone cannot tell us how much memory a run needs.
The weights fit but the request failed
Our Gemma 4 31B run returned HTTP 200 on its first API request, but every label had the same probability. The requests after that returned HTTP 500. This was the development run recorded in issue #4: Ardana 0.2.2 on an M1 Max with 64 GB of memory.
We built Ardana to return typed decisions from local models. An equal probability for every label is still a result a caller can consume, especially when the request reports success. Here, that response was the first sign of a failure.
The affected configuration gave Gemma’s sliding-window layers a cache sized for 40,960 tokens. Of the model’s 60 layers, 50 have an attention window of 1,024 tokens. We were allocating a cache much larger than that window, which caused this failure.
Weights are only part of the allocation
Our catalog maps Gemma 4 31B to Q4_0 and a file of 17,992,313,088 bytes. That identifies the catalog artifact. It does not establish the quant used in the recorded run, because the report never names it. I cannot use that download size as the memory budget for inference.
In the recorded comparison, we had 17,144 MiB of weight buffers and 2,132 MiB of compute buffers. The KV cache was a separate allocation for attention state. Those buffers need memory alongside the weights. Adding the components gives an allocation sum, not a measurement of total process memory.
Our default load uses a 40,960-token context and offloads all model layers. A short prompt does not change that configured capacity. We check the actual prompt length against the context we loaded.
When I compare whether a model fits, I need the runtime configuration beside the file size. That comparison needs the quant, layer placement, number of loaded models, and available device memory alongside context capacity. The download size leaves most of that out. In this case, the cache allocation was the part to inspect.
Sliding attention was getting a full cache
We inherited a default that llama.cpp’s own command-line tools do not use. Ardana uses llama-cpp-2 version 0.1.159, whose bundled llama.cpp context API defaults to full storage for sliding attention. The common parameters used by its command-line tools default to bounded storage instead. Our context followed the API default, so the sliding layers received storage for the entire configured context.
With bounded storage, the allocator takes the smaller of context capacity and the sliding-window capacity plus one micro-batch. It then pads that capacity to 256 cells. For our single-sequence configuration, a 1,024-token window plus a 2,048-token micro-batch gives 3,072 sliding-cache cells instead of 40,960. Those cell counts come from the allocator’s capacity calculation.
Full storage has a purpose: it lets a sequence rewind beyond the local attention window. We clear the KV cache before every prompt, and our decode lifecycle never rewinds. We were reserving memory for an operation we do not perform. The bounded cache reuses old cells only after the model’s attention window masks them.
I fixed that allocation by adding one explicit setting to our context construction. The configured input window and model weights stay the same. Our batch capacity still equals the context capacity, and the micro-batch capacity remains the smaller of context capacity and 2,048 tokens. Full-attention layers keep their separate full-context cache; this setting bounds storage for the sliding layers:
.with_swa_full(false)The cache fell from 35,200 to 5,600 MiB
Our comparison in issue #4 records the allocation change in our development configuration. I use its integer figures here, with all table values in MiB.
| Allocation or limit | Before | After |
|---|---|---|
| Model weights | 17,144 | 17,144 |
| KV cache | 35,200 | 5,600 |
| Compute buffers | 2,132 | 2,132 |
| Sum of allocations | 54,476 | 24,876 |
| Metal limit | 53,084 | 53,084 |
Only the KV cache changes in this comparison. We kept the weights and compute buffers at the same sizes. Reducing the cache moved the sum of these three allocations from above the recorded Metal limit to below it. This sum does not measure total process memory. I use the Metal limit as a separate reference for comparison, not as part of the allocation sum.
Our runtime documentation breaks down the cache reduction: sliding layers fell from 32,000 to 2,400 MiB. The other layers kept their 3,200 MiB cache. Full-attention layers still retain separate storage for the full context. In the sliding layers, cells become reusable only after the attention window masks them.
We reduced cache allocation without shortening the input window or changing the model weights.
We shipped the cache fix in Ardana v0.3.2 on October 13, 2026. The change preserves the configured input window, model weights, and batch settings. I keep the unchanged rows in the table so we can see the scope of the fix alongside the smaller cache.
A failed decode must not look uncertain
We also needed to stop a failed decode from looking like uncertainty. Metal’s failed asynchronous work could leave the decode call reporting success. Synchronization then yielded an untouched, zero-initialized logit row, and softmax converted those zeros into equal label probabilities. Our caller saw an answer even though the GPU had failed to produce logits.
The new guard rejects a slot when every selected label logit equals zero. We catch that untouched row before it becomes a probability distribution. The error makes the failure explicit:
llama.cpp returned no logits: the GPU decode failed, most likely out of memoryOur guard catches untouched logits on a context’s first failed decode. After an earlier successful decode, a failure can leave stale nonzero logits instead. The zero check cannot detect those, so we still do not have comprehensive GPU-failure detection.
In PR #15, we recorded eight successful Gemma 4 31B requests on an M1 Max with 64 GB. Those included prompts of 5,018 and 12,238 tokens. Forcing full-cache allocation back caused the first request to return HTTP 500 with the new guard. These runs do not establish the accuracy of the model’s decisions.
Check the context you need
I start with the update and version checks below. The updater handles installer-managed copies. For unmanaged or source builds, it prints the installer command.
ardana update
ardana --versioncurl -fsSL https://ardana.ai/install.sh | shFor shorter prompts, I can set an 8,192-token context. We checked this example against the CLI syntax without executing it. Prompts exceeding the loaded window are refused, not truncated.
ardana run gemma-4:31b "Please refund the duplicate charge." --noul "Does the customer request a refund?" --n-ctx 8192 --jsonThe context and GPU-layer flags apply only to the run command. Our server exposes neither flag and uses the 40,960-token default with all layers offloaded. It defaults to one loaded model, with a configurable limit. Changing a local run’s context does not change those serving defaults.
ardana serveWe silence llama.cpp allocation logs in the normal binary. Verbose run output reports the model, tokens, and timings; it does not enable those allocation logs. For the engineering evidence behind the fix, see the recorded request tests in PR #15.