Budget tokens for local decision requests

A state length is not a request budget

I can count every token in my state and still underestimate an Ardana request. The model reads a rendered row containing the state, question instructions, options, and answer prefix. Nonempty option descriptions add prompt text too. Our prompt builder encodes the context and question pieces separately, so exact counting has to preserve those boundaries.

Then I have to account for repetition. On Ardana’s causal-model path, each question becomes a row, and isolated scoring creates a separate yes/no row per level. In v0.3.2, we decode each row separately, and the llama.cpp backend clears its KV cache before each row. Asking more questions means processing the same state again.

Our usage accounting reports input usage by counting the shared context and common suffix prefix once. That count cannot generally replace either the longest complete row or the sum of all complete rows. I need the longest row to check whether each row fits, and the sum to check the request’s aggregate budget.

Budget the rows the model reads

I budget the complete scoring rows our causal-model planner produces. In v0.3.2, the runtime limits default to 1,024 rows, 36,864 tokens per row, and 1,048,576 tokens summed across the request. The configured runtime context defaults to 40,960 tokens. With those defaults, I check all three admission conditions:

rows <= 1024
max(row_tokens) <= min(context_window, 36864)
sum(row_tokens) <= 1048576

The context in that check is the capacity available to the runtime. Planning rejects oversized requests before loading the model, and inference checks again against the loaded runtime’s actual context. Raising context leaves the per-row and aggregate caps in place, so I check those separately.

Rows also multiply before we reach inference. Our planner makes one row per question, but an isolated score question expands into one yes/no row per score level. Switching score isolation off changes the question to listwise evaluation. I treat that as a change in score interpretation, because it changes what we ask the model to evaluate.

Reported input usage counts shared context and the common suffix prefix once. That accounting does not cache execution: in v0.3.2, we decode each causal-model row separately, and llama.cpp clears its KV cache before each row. Additional questions therefore process the state again, while the aggregate limit counts those complete repeated rows. I need the longest row and the sum of all rows to judge admission; the usage total cannot stand in for either.

Measure a real ticket before increasing context

I measured a support ticket on Ardana v0.3.2 before changing the context setting. The customer reports a duplicate charge and asks for a refund. We ask which department should handle it and whether the customer requests a refund. Save this exact body as request.json so we can repeat the measurement with the same state and questions.

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		},
		"refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
	}
}

With Ardana installed, we can check the binary version and fetch the model named in the request. The binary I ran matched the release above. Keep that version alongside the request when comparing results, because these measurements belong to a specific model and runtime. For a fresh setup, we install Ardana with the documented command below.

curl -fsSL https://ardana.ai/install.sh | sh

ardana --version
ardana pull decider:2b

Next I ran the saved request with a context setting of 8,192 tokens. The command reads the body from the file and requests JSON output. Verbose mode also prints token counts and timings to stderr. We can keep that diagnostic stream separate when capturing the JSON response.

ardana run decider:2b --request request.json --n-ctx 8192 --json --verbose

The run succeeded and reported 72 input tokens and zero output tokens. Below is the observed usage excerpt from that response. I keep those counts attached to this ticket and model; changing the input gives us another request to measure.

{ "usage": { "input_tokens": 72, "output_tokens": 0 } }

Then I repeated the command with the context reduced to 16 tokens. The saved request stayed identical. This run exited with status 1 and printed the refusal below.

ardana run decider:2b --request request.json --n-ctx 16 --json --verbose

error: request too large: the state and question need 49 tokens, more than the model's context window of 16 tokens

The refusal identifies a 49-token requirement for the state and question. I would repeat this check with the actual application request before choosing its context setting.

Separate capacity failures from useful context

I used the same saved ticket to check the local HTTP path. The isolated server returned HTTP 200 with the same reported usage as the CLI run. Start the server in one terminal and send the request from another.

ardana serve --port 18832

curl -sS -w '\nHTTP %{http_code}\n' http://127.0.0.1:18832/v1/systemone -H 'Content-Type: application/json' --data-binary @request.json

Next I replaced the state with 38,000 repetitions of a space followed by “word” and kept only the refund question. This deliberately crosses the default per-row budget. I ran the commands below; the output shows the response body and status I observed.

jq '.state = (" word" * 38000) | .questions = {refund:.questions.refund}' request.json > over-row.json
curl -sS -w '\nHTTP %{http_code}\n' http://127.0.0.1:18832/v1/systemone -H 'Content-Type: application/json' --data-binary @over-row.json

{"detail":{"error_type":"request_too_large","message":"too many tokens: one row has 38029 tokens, the limit is 36864 per row"}}
HTTP 413

This refusal came from a row containing 38,029 tokens against the 36,864-token default limit. I also tested 35,000 repetitions with 31 refund questions: each row fit, but their sum reached 1,085,744 tokens and exceeded the default 1,048,576-token request budget. The server separately caps the HTTP body at 32 MiB, a byte limit rather than a token count. Causal-model planning rejects oversized requests before model loading and supplies no state truncation cap.

I treat an accepted request as a capacity result, not proof that the model used its evidence well. Lost in the Middle tested multi-document question answering and key-value retrieval, finding performance often strongest with evidence near the beginning or end and weaker in the middle. RULER extended retrieval tests with multi-hop tracing and aggregation and reported degradation as context grew. Those findings concern the models and tasks in those studies; they give us a reason to evaluate evidence position and input length, not an Ardana quality measurement.

Keep a budget regression case beside the request

I would keep the ticket request as a budget regression case, with expected decisions beside it. With the model, questions, labels and distractors fixed, I would move the same evidence between the beginning, middle and end. In a separate pass, I would vary input length and record probabilities, decision errors and refusals. We have not run this quality experiment on Ardana.

When reducing an input, I would preserve the decision rules and the evidence needed to apply them. Repeated background text is a better candidate to cut, provided the expected answer stays justified. I would split questions only when each is meaningful on its own. Each causal-model row processes the state again, so splitting a request does not remove that repeated work.

For each version, I would check the longest rendered row and the sum across rows alongside decision quality. The planner can expose those counts without inference. That gives us a capacity check to repeat when a request changes. Adapt the ticket fixture to your own decisions and inspect the planner in Ardana’s repository.