Cache local decision results without stale answers

A repeated sentence is not the same decision

I do not trust a cached decision just because the same sentence appears again. If the question or the evidence behind it has changed, yesterday’s answer can be wrong today. Before I reuse an answer, I need to know what stayed the same.

HotKV’s recent cache experiment makes me ask whether a cache hit gives us an answer that still applies to our local decision. I start with what we are caching.

With exact result caching, I reuse a response when keys match. With semantic caching, I accept an embedding similarity threshold, which can match requests that mean different things. A transformer KV cache stores attention keys and values for further computation.

In Ardana v0.3.2, we reuse readers and loaded workers, while our HTTP handler sends requests through inference without a completed-response lookup. Our llama.cpp backend also clears its KV cache before decoding each prompt. To reuse a completed answer, I need to define what makes the whole request identical.

Preserve the request before building a key

I keep the complete request when I build an exact cache key. The HTTP body requires a state and a questions object; model selection can be absent, null, or a string. Save this documented noul request as request.json:

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
	}
}

Questions retain insertion order and answer IDs. I preserve both in the key, along with the full instructions. State strings pass through unchanged, while structured states retain object iteration order and have special array rendering. Choice option order and descriptions enter the rendered question too. Score and noul questions have their own ordering rules, so I keep the original request instead of assuming these transformations are interchangeable.

For scores, I also preserve the isolation setting. Together with the model profile, it changes eligible score readout planning. Dropping that setting from the key can make different computations share a cached answer.

The returned model name is the profile name. I cannot use that label to verify the actual deployment behind an answer. I can inspect the registry configuration with:

ardana show decider:2b --json

That output includes weights and tokenizer paths, runtime, and profile, but no content checksum. Hashing it alone cannot detect changed bytes at unchanged paths. Profiles also include layout, temperatures, and readout configuration. I therefore require a verified fingerprint covering actual weights, tokenizer, profile, and runtime/build configuration for reuse. Establishing that identity is our responsibility as callers; it is not an Ardana API feature.

Bind the cache to its dependencies

I put the complete serialized request inside a versioned identity envelope before hashing it. Alongside it, I include the tenant, deployment revision, evidence revision and output contract revision. These belong to our cache wrapper and stay outside the HTTP request body. A matching request is reusable only when those dependencies still match.

Our deployment revision must identify the weights, tokenizer, profile and runtime/build configuration in use. Ardana’s profile configuration includes layout, temperatures and readout settings, so weights alone are insufficient. I also need our revisions to account for changes in preprocessing, evidence and the output contract. As callers, we must bind these revisions to their dependencies. Ardana’s API does not do this for us.

This Python excerpt shows the key construction, not a complete cache. I include the standard-library imports and a frozen identity dataclass with four required string fields. Our encoder preserves dictionary insertion order, rejects nonfinite floating-point values and produces UTF-8 bytes. I encode the request, then encode an envelope containing its decoded JSON string and the identity’s field/value pairs.

import hashlib
import json
from dataclasses import asdict, dataclass


def encode(value):
    return json.dumps(
        value, ensure_ascii=False, allow_nan=False,
        sort_keys=False, separators=(",", ":"),
    ).encode("utf-8")


@dataclass(frozen=True)
class Identity:
    tenant: str
    deployment_revision: str
    evidence_revision: str
    contract_revision: str


def key_and_body(request, identity):
    body = encode(request)
    envelope = ["ardana-result-v1", asdict(identity), body.decode("utf-8")]
    return hashlib.sha256(encode(envelope)).hexdigest(), body

On a miss, I send the returned request bytes unchanged to the local inference endpoint, without serializing the request again. This keeps the bytes represented in our key tied to the bytes we send. Complete semantic identity means accounting for every relevant dependency. The lookup still requires an exact key match.

Changing a revision makes our next lookup miss, but it does not delete an old entry. I still need explicit purging and bounded retention for those entries. This example is sequential and process-local, and it cannot discover deployment or evidence changes automatically. Our revision checks establish identity; expiration needs its own checks.

Expire answers and reapply the action policy

I check caller-supplied evidence validity before looking up an answer and again after inference finishes. Evidence can expire while we wait for the result. I cap each entry’s expiry at the earlier of its TTL deadline and the evidence deadline. Reading an entry never extends that deadline.

There is no universal TTL. I choose ours based on how often the source changes and how much staleness the application can tolerate. Expiration bounds reuse; it does not discover an upstream update. Our caller still has to supply current evidence revisions and validity.

My example keeps a bounded number of entries and evicts the least recently used entry when it needs room. That bounds entry count, not memory in bytes. Changing a revision causes a miss but leaves old entries behind, so I keep an explicit purge operation too. This cache is sequential and process-local, with no automatic deployment or evidence discovery and no mechanism to combine concurrent misses.

Before storing anything, I validate the successful response. This example supports noul questions only: I check that answer IDs match, answer types are correct, and probabilities are finite and between 0 and 1, inclusive. Failed inference or invalid responses never enter the cache. I retain decision data without historical usage, because those token counts describe the original inference.

Suppose we cache a probability of 0.75 and then change our action threshold from 0.70 to 0.80. The same probability passes the old threshold and fails the new one. I can reuse the probability while the request and its dependencies remain valid, but I must apply the current threshold. Storing the action alongside it would preserve a policy decision we have already changed.

Try one request before adding reuse

I start with a direct request before putting a cache in the path. Save the complete JSON body from earlier as request.json in your working directory. With Ardana installed, I pull the model and start the local server.

ardana pull decider:2b
ardana serve

Leave the server running and open another terminal in that directory. I send the saved body straight to the local endpoint. This is the request I want working before adding reuse.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

I have not measured workload hit rates or latency savings for this design. Cache mechanics alone do not tell me what an application will save. I would measure that on the requests the application actually sends.

Before enabling reuse, I still have to bind the cache identity to the actual deployment and current evidence. A model name cannot establish that binding for me. Start with Ardana on GitHub, run the request, and decide which dependencies your application must track.