Benchmark local decision API latency under load

Time the complete decision response

I cannot judge a local decision API by its token generation speed. I need the complete answer before an automation can use it. In Ardana, our response contains typed answers and usage, with reported output tokens set to zero:

output_tokens: 0

My measurement unit is one complete decision response. I stop the client timer after reading the full response body, then validate the answer before counting a successful completion. That is the response-reading boundary our playground uses, though validation remains a separate step in this proposed protocol. A successful HTTP status alone does not establish that the body is correct.

Our HTTP handler exposes no server duration field or server timing instrumentation. Client elapsed time therefore cannot tell me how much time the server spent on each internal step. The CLI has its own loading, answering and total durations, but those measure a different execution path. I cannot subtract them from an HTTP measurement to recover server timings.

I am proposing a measurement protocol here. I have no benchmark results to report. The first distinction I need is between loading a model and serving requests while it stays resident.

Separate model loading from warm serving

Before serving, I would record the installed version, pull the model, and save its metadata. The source behavior here is pinned to v0.3.2; this is a proposed setup, with no measured timings. Pulling first keeps downloading separate from the first decision request. The metadata records which model we pulled.

ardana --version
ardana pull decider:2b
ardana show decider:2b --json

With the model provisioned, I would start the server in another terminal. These environment controls apply to the serving process after provisioning: they disable library access and background update checking. The explicit keep-alive gives our warm sequence a declared residency window.

ARDANA_LIBRARY=off ARDANA_NO_UPDATE_CHECK=1 ardana serve --default-model decider:2b --keep-alive 10m

For each cold-loading run, I would restart the server and measure its first decision separately. That request includes model resolution, tokenizer preparation, planning, and loading when necessary before the queued work completes. I would label this process/model cold loading, because restarting does not establish an empty OS page cache. Then I would warm the server with successful requests and measure the following sequence within keep-alive.

The server defaults keep weights alive for 5 minutes and allow 1 loaded model. Idle expiry or loading another model can unload those weights, so I would keep the same model resident throughout the warm sequence. Before measuring, I would check the loaded-model field exposed by the health endpoint shown below. That check neither performs inference nor reports queue duration; it cannot replace the successful warm-up requests.

GET /health
x_loaded

Keep the client timer and workload visible

I would keep the request fixed while changing the client workload. That makes the state and question part of the benchmark definition. Save this reduced request as request.json so every attempt asks the same refund question.

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"refund": { "type": "noul", "instructions": "Does the customer ask for a refund?" }
	}
}

For a complete-transfer sample, I would use the proposed command below. Before counting a response as successful, I would preserve the command’s exit status and inspect the saved body for the expected model, answer type, numeric refund score and usage.

curl -sS --max-time 60 -o response.json -w 'HTTP %{http_code} seconds %{time_total}\n' http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' --data-binary @request.json

The curl timer measures the whole transfer in seconds; first-byte timing stops before the complete decision arrives. Separate curl processes cannot reuse connections. I would declare connection reuse in the load client, consume the full body before stopping its timer, and state whether JSON parsing and validation are timed. Neither HTTP timer isolates queueing or inference.

For a local diagnostic, I would run the CLI with verbose timing. Its load duration surrounds model loading, and answer duration surrounds answering after planning, including decoding and readout. Total duration excludes process startup and input reading, then covers resolution, possible pulling, planning, loading, answering and teardown through printing the answer. Each invocation loads anew, so I cannot subtract these durations from resident HTTP latency to recover server phases.

ardana run decider:2b --request request.json --json --verbose

I would also record questions and scoring rows alongside request counts. On the causal path, each question becomes a row, with isolated scoring expanding it per level when enabled. Reported input tokens count shared prefixes once, while execution decodes rows separately. Our context budget guide covers admission; here, those different counts explain why reported input-token throughput does not describe all executed row work.

Increase arrivals without hiding the queue

While increasing arrivals, I would keep the request fixed because our server has one FIFO worker per model. That worker executes one request at a time, without batching requests together. Admission counts waiting and executing scoring rows across all models; with a limit of 4096 rows, exceeding that allowance returns HTTP 503. Increasing the loaded-model limit does not add workers for the same model.

For the first sweep, I would hold concurrency fixed, with each client waiting for completion before starting its next request. This is a closed loop: as responses slow, clients send less work. Grafana explains the resulting coordinated omission: the test can hide demand that would arrive independently of response times. It tells me how a fixed client pool behaves, but leaves that arrival pattern untested.

Then I would use an open loop with scheduled arrivals, so starts do not depend on completions. The generator can still run out of available clients, so I would report scheduled arrivals, actual starts, and dropped iterations separately. Work that never starts is not an HTTP failure. I would also record how late each actual start is relative to its schedule.

MLPerf Server uses Poisson arrivals and measures latency from scheduled submission through reply receipt. I would keep that interval alongside dispatch-to-response time, with scheduling delay recorded separately. Following this measurement idea does not make our harness MLPerf-compliant. I have not executed this load test, so the queue’s effect on latency and throughput remains to be measured.

Report successful decisions within the deadline

I would calculate p50, p95 and p99 only from successful requests with validated response bodies. HTTP 200 alone does not qualify a response. Checking the expected structure and answer types establishes that we received a usable response. It does not establish semantic accuracy: a well-formed answer can still be wrong.

For those percentiles, I would publish the successful sample count, run duration and quantile estimator. A tail estimate needs that context, especially with few observations. Pool the observations across workers before calculating each percentile. Averaging worker percentiles loses the distribution we need, as the Prometheus histogram guidance explains.

Every attempted request stays in my accounting. Report HTTP errors, transport failures, invalid bodies and timeouts separately, with each count divided by all attempts. Report the overall failure fraction with that same denominator. Keep timeouts out of completed-decision latency samples; a timeout tells us when waiting stopped.

For a finite run, I would divide validated successful completions by elapsed seconds from first dispatch to final terminal outcome. That interval includes waiting for the last failure or timeout. Deadline attainment is successful completions within the declared deadline divided by all attempts, following the latency SLI approach.

My next step is to record the run manifest before collecting measurements. Put the request, load schedule, timing boundary, validation rule and deadline in it so we can repeat the run. Then inspect the worker in Ardana’s source alongside that manifest.