Handle local decision API failures

No answer is different from an uncertain answer

If I replace an inference exception with a probability, I have invented an answer. A numeric fallback keeps the application moving, but it hides whether the model answered at all. The next threshold check sees an ordinary score. It can no longer distinguish uncertainty from a failed request.

Our server contract test accepts HTTP 200 with this answer fragment. This is synthetic test output, not a fresh model response. The probability is 0.5, and that uncertainty belongs in the success path:

{ "type": "noul", "noul": 0.5 }

Uniform probabilities are also valid when the selected logits are equal, finite, and nonzero. A uniform choice has zero confidence and still names the first tied option. I can use that result to abstain, but I still have an answer to evaluate.

Reported runtime failures return HTTP 500, which means I have no answer to evaluate, not a low-confidence answer. If I turn that failure into a score, the threshold check evaluates my fallback instead of the model’s result.

Check the response contract before the threshold

I start by pulling the model and starting the local server. The first command prepares the weights. The second serves the decision API.

ardana pull decider:2b
ardana serve --default-model decider:2b

For this example, I save the source-derived request below as request.json in the current directory. It asks whether the customer wants a refund. We require the input text and question definitions; specifying a model is optional.

{
	"model": "decider:2b",
	"state": "Please refund the duplicate charge.",
	"questions": {
		"refund": {
			"type": "noul",
			"instructions": "Does the customer ask for a refund?"
		}
	}
}

With the server running, I send that file from another terminal. The command reads the file as the request body and selects HTTP POST. This checks the request wiring before I add the application’s decision logic.

curl http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' -d @request.json

In JavaScript, I check the HTTP status before treating the body as an answer. A resolved fetch promise can still carry an HTTP error. JSON parsing can fail too, and cancellation can interrupt body reading after the headers arrive. I keep each of those failures out of the threshold comparison.

After parsing, I validate the response shape, including model identification, typed answers, and usage information. I look up the answer by the exact question ID I sent and check that its type matches the request. For this refund question, I require a finite numeric yes probability between zero and one, inclusive. Our runtime checks do not comprehensively validate finite values, so I make that a caller check. Only an answer that passes these checks reaches the threshold.

A valid schema still does not authorize a refund. I use this question to classify the customer’s request; permission to issue money remains an application rule. When the server rejects an input, I keep that rejection as an error and inspect its cause before deciding what happens next.

Change invalid inputs instead of retrying them

I handle a validation failure by changing the request before sending it again. An unchanged request carries the same defect into the next attempt. Ardana returns HTTP 422 with an array of validation errors, so I keep that structure available to the caller. This is an illustrative, source-derived response for a missing required input:

{
	"detail": [{ "loc": ["body", "state"], "msg": "Field required", "type": "missing" }]
}

For an invalid question, the location identifies the request body, the question collection, and the question ID. Malformed JSON also gets 422, with a character offset and parser context. An unreadable transport body gets 400, so I keep that failure separate from an invalid question.

Request size failures use a different shape: HTTP 413 carries an error object. I check the status before choosing how to read the error payload. This oversized-body response is also illustrative and source-derived:

{
	"detail": {
		"error_type": "request_too_large",
		"message": "the request body is larger than 32 MiB"
	}
}

The body limit is 32 MiB, but we also limit the work inside the request. The defaults allow 1,024 scoring rows, 36,864 tokens per row, and 1,048,576 tokens summed across rows. We check the context window separately, and isolated score levels can expand one question into multiple scoring rows. Planning rejects oversized inputs without truncating them, before the weights load. I reduce the input or requested scoring work before submitting it again.

I correct credential and model selection errors before retrying. Missing credentials return 403, wrong credentials return 401, and an unknown model returns 404. More time in a queue will not fix those requests. I reserve overload handling for capacity problems, with separate recovery paths for input validation, authentication, and model selection.

Keep overload retries bounded

Ardana admits work by scoring rows across all models. We count both waiting rows and rows currently decoding against the same budget. Each model has one FIFO worker processing one request at a time. I can configure that budget with this command:

ardana serve --max-queued-rows 4096

Overload returns HTTP 503. I read the retry header before scheduling another attempt, because its value here means wait one second. The header gives us a delay before a follow-up, not a promise that capacity will be available:

Retry-After: 1

Waiting cannot help a request that exceeds our configured row budget. It gets 503 even when the queue is empty. I have to reduce the work in that request or raise the budget before it can fit.

For temporary overload, I cap both the attempt count and total elapsed time. I respect the server’s delay and use capped exponential backoff with jitter before further attempts. We keep retry logic at one layer, since retries can amplify overload. Once that budget is exhausted, I return the failure to the caller.

Cancelling the client wait does not guarantee we release server capacity. Once submitted, an abandoned request can keep decoding and hold its row reservation until completion or discard. I keep the deadline active through response-body reading too, because cancellation can reject that read after headers arrive. Model keep-alive controls idle unloading; it does not set our request deadline.

Downstream actions stay outside my inference retry loop. A timeout does not prove an external action failed. If we retry that action, it needs idempotency so repeating the attempt does not repeat the effect.

Keep failed inference visible to the caller

We covered Gemma cache allocation in an earlier post. In Ardana v0.3.2, we reject a slot when its selected label logits are all exactly zero. The caller gets HTTP 500 with this message from the source:

llama.cpp returned no logits: the GPU decode failed, most likely out of memory

That guard misses stale nonzero logits left after an earlier successful decode. I cannot treat it as comprehensive GPU failure detection. Our health endpoint does not run inference or expose the queue counter. A passing health check therefore cannot prove that the next decision will succeed.

At the application boundary, I would persist the failed request as unresolved and keep its error available to the caller. That is an application design choice, separate from the server’s response contract. It would stay unresolved across restarts until a later attempt produces a validated answer or someone handles it explicitly. An invented fallback score would hide why the request stopped and let it reach the action threshold as if inference had completed.

For a concrete client test, inject HTTP 500 and assert that no score is created and no action runs. Restart the application and check that the request remains unresolved with its error intact. Start with the error contract in Ardana’s repository and add that test beside your successful decision test.