Shadow-test local decision models before rollout

Let the candidate answer without letting it act

I want to upgrade a ticket-routing model without letting the first comparison reroute real tickets. The current model is my baseline. I would freeze each request and give both models identical copies. Only the baseline response would be allowed to affect production.

AWS documents this pattern in its shadow testing guide: copy inference requests to a candidate and return only the production response. I would implement the same pattern in application code around Ardana. Ardana takes state and typed questions and returns probabilities. The application owns the action that follows.

I would make the action boundary explicit: only the baseline result would reach the action executor. The candidate worker would have no production write credentials or tools, and its output would go to comparison storage.

Agreement alone would not convince me to promote the candidate. Two models can choose the same wrong route, and a disagreement does not tell me which is right. I would review both agreements and disagreements against independent labels before changing the model allowed to act.

Start two explicitly configured local endpoints

I would use decider:0.8b as the baseline and decider:2b as the candidate. Their default quantizations are q8_0 and q4_k_m, respectively. Those roles do not establish which model makes better decisions for our task. I pull both models before starting either service. This setup assumes free ports, sufficient resources, and an unset API-key environment variable.

ardana pull decider:0.8b
ardana pull decider:2b

For the baseline, I open one terminal and set its default model explicitly. The environment settings disable library refreshes and update checks after preparation. They do not guarantee offline operation: a missing named model can still be pulled on first use.

ARDANA_LIBRARY=off ARDANA_NO_UPDATE_CHECK=1 ardana serve --host 127.0.0.1 --port 8000 --default-model decider:0.8b --max-loaded-models 1 --max-queued-rows 1

In a separate terminal, I start the candidate on port 8001. I leave model selection out of the request so each server uses its configured default. That lets us send identical request bytes to both processes while comparing different models.

ARDANA_LIBRARY=off ARDANA_NO_UPDATE_CHECK=1 ardana serve --host 127.0.0.1 --port 8001 --default-model decider:2b --max-loaded-models 1 --max-queued-rows 1

I cap admission at one scoring row per server, including unfinished work. The request below has one choice row; excess admission receives an overload response. This limit reserves no CPU or RAM, and separate ports still share hardware. A client deadline does not release the row while admitted model work is still running.

Before timing the replay, I warm each endpoint with an actual valid inference request. Models load on inference, so a successful health check alone does not prove readiness. I check that both requests succeed before collecting timed pairs.

Freeze the request before either branch sees it

I would freeze the state, questions, options, and action policy before either branch can change them. For this replay, I save the following as example-request.json and read its bytes once. Both endpoints receive those same bytes, with the model omitted so each uses its configured default. I keep the pairing ID outside the request because putting it in the state would change the model input.

{
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		}
	}
}

We need Python 3 and curl for this small harness. I save it as shadow-pair.py; it assigns one UUID, starts two requests, and keeps each result under its branch name. Both threads share the immutable payload. The harness has no retries or action executor. Errors stay in their records instead of becoming default probabilities.

from concurrent.futures import ThreadPoolExecutor
import json
import math
from pathlib import Path
import subprocess
import sys
import uuid


def valid_choice(data, options):
    try:
        answer = data["answers"]["department"]
        probabilities = answer["probabilities"]
        return (
            answer["type"] == "choice"
            and isinstance(probabilities, dict)
            and set(probabilities) == set(options)
            and all(type(p) in (int, float) and 0 <= p <= 1 and math.isfinite(p)
                    for p in [*probabilities.values(), answer["confidence"]])
            and answer["choice"] in probabilities
            and probabilities[answer["choice"]] == max(probabilities.values())
            and abs(math.fsum(probabilities.values()) - 1)
                <= len(options) * 0.00005 + 1e-12
        )
    except (KeyError, TypeError, ValueError, AttributeError):
        return False


def call(port, payload, options):
    command = ["curl", "--noproxy", "*",
               "--max-time", "5", "--header", "content-type: application/json",
               "--data-binary", "@-", "--write-out", "\n%{http_code}",
               f"http://127.0.0.1:{port}/v1/systemone"]
    result = subprocess.run(command, input=payload, capture_output=True)
    body, _, raw_status = result.stdout.rpartition(b"\n")
    http_status = int(raw_status) if raw_status.isdigit() else None
    record = {"curl_exit": result.returncode, "http_status": http_status,
              "body": body.decode(errors="replace")}
    if result.returncode:
        record["status"] = "deadline" if result.returncode == 28 else "transport_error"
        record["error"] = result.stderr.decode(errors="replace")
    elif http_status != 200:
        record["status"] = "http_error"
    else:
        try:
            data = json.loads(body)
        except (ValueError, UnicodeError):
            data = None
        record["status"] = "ok" if valid_choice(data, options) else "invalid_response"
    return record


if __name__ == "__main__":
    payload = Path(sys.argv[1]).read_bytes()
    options = json.loads(payload)["questions"]["department"]["criteria"]
    request_id = str(uuid.uuid4())
    with ThreadPoolExecutor(max_workers=2) as pool:
        baseline_future = pool.submit(call, 8000, payload, options)
        candidate_future = pool.submit(call, 8001, payload, options)
        baseline = baseline_future.result()
        candidate = candidate_future.result()
    print(json.dumps({"request_id": request_id, "baseline": baseline,
                      "candidate": candidate}, indent=2))

I use a five-second transfer deadline to keep this example bounded. It is illustrative, with no model inference measurements behind it. The records keep deadline, transport, HTTP, and malformed-response failures separate and preserve the response body and HTTP status. I reserve ok for responses that pass the client admission checks I wrote for this example. Those checks allow four-decimal probability rounding, but do not verify confidence formula consistency, establish native API conformance, or fully validate responses. I verify the installed binary and server flags before the replay.

ardana --version
ardana serve --help
python3 shadow-pair.py example-request.json > pair.json

I wait for both replay records before writing the file so I can inspect them offline. This is not a production mirror. The synthetic checks behind the accompanying source exercised identical request bodies, pairing, and failure handling, but no actual inference or rollout occurred. Before I compare choices, I need to decide which recorded answers qualify as valid pairs and keep unavailable candidates in the accounting.

Count unavailable candidates before comparing choices

I would count all selected snapshots in N, including requests the candidate never answers. P counts pairs where both responses are valid and arrive within the deadline. F counts unavailable candidate responses, including queue drops, deadlines, transport failures, HTTP failures, and malformed responses. D counts those valid pairs whose proposed actions differ.

Coverage is P/N, candidate failure rate is F/N, and disagreement is D/P. A zero denominator makes that rate undefined. If only the baseline fails, coverage falls without adding a candidate failure.

Before admitting a pair, our replay code validates the choice fields it consumes, not the full response schema. I require the expected answer type, a chosen request criterion, and numeric, finite confidence. The returned model name identifies a profile that may differ from the configured name. It does not give us full provenance. Probabilities must be a dictionary with exactly the requested criterion keys and finite, nonboolean numeric values in [0,1]. I check the sum below, allowing half a four-decimal rounding unit per option plus a floating-point allowance, without renormalizing.

abs(math.fsum(probabilities.values()) - 1) <= len(options) * 0.00005 + 1e-12

Latency and availability tell me whether a candidate answers in time. They do not tell me whether its answer is correct. I would use independent labels or review on disagreements and agreements, because both models can share a mistake. An unexecuted candidate action gives us no observed outcome for that action.

For a live mirror, I would use a bounded, nonblocking candidate queue and record every drop. The baseline must never wait for candidate completion. A client deadline does not cancel admitted server work: its rows still count until the worker finishes. A timed-out request can therefore continue occupying admission capacity.

Write the promotion and rollback rule first

Before mirroring traffic, I would write down the request population, required case coverage, deadlines, and resource budgets. I would also define the quality and failure criteria and the triggers for rollback. Only the current model’s results would reach the action executor until the application switches models. The candidate would have no production write credentials or tools.

Separate ports do not isolate CPU, RAM, or accelerator use. For a live mirror, I would enforce those limits outside Ardana and keep the candidate queue bounded and nonblocking. The baseline would never wait for it. I would stop mirroring when the declared baseline interference threshold is crossed; a client deadline does not cancel admitted model work.

Promotion would be an application decision. I would retain the old endpoint, model, and policy so the application can switch back when a rollback trigger fires. Ardana has no promotion command to perform that switch.

I have not run real inference, a resource-isolation trial, or a rollout here. The checks exercised the replay against synthetic HTTP responses. To try it, install Ardana from the repository and compare one frozen request against the two configured local defaults. Inspect the shipped server code there before wiring the comparison into your application.