Blog

Record what produced a local model decision

Preserve requests, model revisions, quantization and runtime settings before an upgrade. Separate a useful audit record from identical inference.

A small brown groundhog stores a plain envelope and matching colored parts in an open wooden box.

A model name is not an execution record

After an upgrade, I want to revisit a decision and check what produced it. A saved answer and a familiar model name leave too much to reconstruct. I need to connect that answer to the files and settings used for the original run.

Ardana returns a profile name as the model identity in its response. That response also carries answers and token usage, but it leaves out the registry identity, artifact digest, runtime version, and execution settings. We need to preserve those details separately.

Thinking Machines demonstrates that batch-dependent floating-point reductions can change inference results even with greedy decoding. I take that as a reason to separate recording a run from promising identical inference. Their demonstration does not establish how an Ardana run will behave.

I want an application audit record that connects an answer to what produced it, starting with the resolved model artifact so we can identify the files behind the name. Keeping that evidence does not guarantee a bitwise identical rerun.

Keep the resolved artifact and its configuration

I would keep the resolved files alongside the model name. In our catalog, decider, decider:latest, decider:2b, and decider:2b-q4_k_m currently resolve to the same default quant. An ordinary hf.co/ reference resolves main before downloading. When pinning a Hugging Face download to a specific commit, I use the full-length hash. For a record I can inspect after an upgrade, I need the resolved revision, quantization, and retained bytes.

Our current decider:2b artifact is hf.co/Mapika/decider-2b-GGUF at ff2e5e687327eda9ac34e9a3ca84d3f400672c87. Its quant is q4_k_m, and the file decider-2b-v11-Q4_K_M.gguf contains exactly 1,274,396,800 bytes. The source checkpoint is hf.co/Mapika/decider-2b at 533964dae8be954c5b5e19fa4948e48408094c1e. I keep those revisions separate because they identify different files.

I start with the pull and inspection commands below. The JSON description includes snapshot paths for the resolved model. It has no dedicated revision, checksum, or binary-version fields, so I would save that information alongside it.

ardana pull decider:2b
ardana show decider:2b --json

The published LFS pointer declares the digest below, but that is publisher metadata. To check which bytes I kept, I would compute a local SHA-256 hash of the retained weight file and record that result separately. Python’s file-digest helper can compute the hash from a binary-opened file to measure its contents.

sha256:b7c132a67934d51c81abc96bb7724800f965ff5a288aed3e1ca7d8bc349c1386

I would retain the tokenizer and decision configuration with the weights. Our library pulls use the pinned GGUF commit and any separate tokenizer commit. For decider:2b, the tokenizer comes from its GGUF snapshot; an explicit tokenizer override changes that selection. Its configuration uses plain layout, temperature 1.145, and per-type temperatures of 1.164 for choice, 1.624 for noul, and 1.124 for score. These temperatures scale readout probabilities rather than sampling generated text.

Store the whole typed request and raw response

Once I have the artifact, I need the request that reached it. Our request contains state and keyed, typed questions, with instructions and criteria. The complete request in the README asks which department should handle the message and whether the customer asks for a refund.

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		},
		"refund": {
			"type": "noul",
			"instructions": "Does the customer ask for a refund?"
		}
	}
}

For a CLI run, I would save that JSON as request.json and pass it to this command. The positional model selects what runs, independently of the model value inside the request. For HTTP, leaving out the model selects the server default. I would record the selection used by the entry point alongside the saved body.

ardana run decider:2b --request request.json --json

I would keep the entire state and question definitions, including their order. Criteria can contain option descriptions and ordered score levels, so a list of labels can leave part of the input behind. A string state stays text, while other JSON is rendered before inference. Arrays with at least eight elements receive indexes in that rendering.

On the response side, I would save the raw JSON before extracting the decision for application logic. Choice answers include a choice, confidence and probabilities; score answers also carry a legend, while the refund answer has its own typed value. The response includes ordered answers and token usage, so I would preserve those alongside any optional answer fields. Returned probabilities have four decimal places and scores have two, which means matching serialized answers cannot establish identical underlying logits.

This is a proposed record owned by the application, not a built-in Ardana export. I would keep the request and response together so we can inspect both sides of a later comparison. The response identifies the profile, but it does not identify the runtime version or execution settings. Those need their own place in the record.

Include the runtime and prompt construction

I would keep the runtime version alongside the saved request. Our CLI exposes it directly. Capture it before replacing the binary.

ardana --version

The context size and GPU layer setting also belong in that record. Our current run defaults appear below. We pin llama-cpp-2 to 0.1.159 and the tokenizer dependency to 0.23.2. I would preserve those dependency versions alongside the binary version when comparing builds.

--n-ctx 40960
--gpu-layers -1

Our decider profile uses the plain layout. Chat profiles have a separate construction step, so I would preserve the template identity when using one. A standalone template takes precedence over the template in tokenizer configuration. The renderer closes recognized thinking blocks and fixes template date functions to 2024-07-26.

We check prompt construction with regression tests for Qwen3.5 head and tail token IDs, Gemma wrappers, Llama and Apertus dates, single-token labels, and tail boundaries. Separate Candle and llama.cpp fixture tests require matching choices and probabilities within 0.1. That tolerance belongs to those fixtures; I would choose an application’s comparison criteria separately. Our CI fetches tokenizer and template fixtures before running the workspace tests:

cargo test --workspace --locked

How far does that take us? The Thinking Machines analysis shows batch-dependent floating-point reductions changing results even with greedy decoding. vLLM documents reproducibility under its scheduling or batch-invariance conditions, with the same hardware and version. PyTorch disclaims reproducibility across releases, commits, platforms, and CPU/GPU execution even with identical seeds.

Those sources describe their own systems and experiments. We cannot use them to establish a reproducibility guarantee for Ardana.

Capture one run before the next upgrade

I would keep an application-owned record that joins the complete request, raw response, resolved model information, runtime version, and the policy revision that consumed the answer. Keep the artifact bytes and a locally computed digest alongside it. This is a proposed record for our integration; Ardana’s model inspection and response each supply only part of it. The inspection includes snapshot paths, but it has no dedicated revision, checksum, or binary-version fields.

Replaying a saved answer lets us check what our policy does with that answer. Running inference again asks the model for a fresh result. I would compare the selected choice and the returned probabilities separately, with acceptance rules tied to the decision our application makes. Ardana’s fixture tests require matching choices and probabilities within 0.1, but that test threshold does not establish a tolerance for our application. Returned probabilities are rounded to four decimal places, so identical saved responses do not prove identical underlying logits.

Before the next upgrade, save one real request as the file used below and retain the output from each command. Our request should include its full state and typed questions. Start with the Ardana repository, then capture the model record, binary version, and response for that request:

ardana pull decider:2b
ardana show decider:2b --json
ardana --version
ardana run decider:2b --request request.json --json