Test interference between bundled decision questions

Another question should not rewrite the original task

I hold one decision question fixed and change a sibling question in the same request. The focal question keeps its wording, options and shared context. I want to compare its probabilities alone and bundled.

The OpenJev discussion on HN asks whether shared KV state exposes sibling questions. I take that as a reason to test, not evidence of an Ardana bug. Reusing shared context or a prefix cache does not itself show that questions attend to one another.

My proposed regression compares full probability vectors by option name, with question IDs and option order fixed. Repeated identical requests give the baseline; shuffling options tests a different change. This comparison says nothing about whether multiple labels are logically independent. I have no measured effects to report.

In Ardana’s public causal-model path, we build one complete prompt row per ordinary question and copy the common context into each. I start there to trace what each backend reads, because usage accounting alone cannot tell me how inference state is shared.

Trace what each backend reads

I am tracing Ardana v0.3.2, not making a claim about every System 1 runtime. In our released causal planner, each ordinary question becomes a complete prompt row. An isolated score level gets its own yes/no row. We put exactly one question in each row and evaluate the rows separately. I inspect the planner at that released revision from an Ardana checkout:

git show 445fbc945b02b641eeaef65a3ae0f0252f571aba:crates/ardana-core/src/engine.rs

We tokenize the common context once and copy its token IDs into every row. The public path does not number multiple questions inside one prompt. Deduplicated input-token accounting does not establish shared KV computation or one forward pass for the bundle.

For llama.cpp, we clear the KV cache before every complete prompt and start positions at zero. Candle also starts each row at offset zero without state from a previous prompt. Its causal decoder uses F32 on CPU or BF16 on Apple Silicon Metal, then performs the label readout multiplication in F32.

In the browser, we give each row fresh feeds, empty past state and zero-based positions. The attention input is an all-ones two-dimensional array. That is not a mask separating sibling questions; those questions have already been placed in separate rows.

I trace THX-01 separately because it uses our decision-model path. It encodes one complete sequence per question. Equal returned probabilities also cannot prove bitwise identity: we round them to four decimal places. We can now hold one question fixed and compare its results across requests.

Keep the focal question fixed across requests

I record the run’s configuration with its results: the actual binary version, model files, profile, quantization, runtime and device. The catalog default has a specific GGUF revision, so a model name alone is not my experiment record. These commands check the version, pull the model and show its configuration:

ardana --version
ardana pull decider:2b
ardana show decider:2b --json

The server runs in another terminal. I keep its runtime and device fixed while sending the variants. This is the process that receives the requests:

ardana serve

I save the complete starting bundle as request.json and use the duplicate-charge state for every request. The department choice is the focal question for the comparison below. The refund and charge-count questions are its two siblings:

{
	"model": "decider:2b",
	"state": "My card was charged twice for order A-104. Please refund the duplicate.",
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which department should handle this?",
			"criteria": ["billing", "technical", "sales"]
		},
		"refund": {
			"type": "noul",
			"instructions": "Does the customer ask for a refund?"
		},
		"charges": {
			"type": "score",
			"instructions": "How many times was the card charged for the order?",
			"criteria": ["Never", "Once", "Twice", "Three times or more"]
		}
	}
}

To establish the alone cases, I send each question separately with that same state. Then I send the complete bundle in its original order and with the question order reversed. I also add the unrelated question below, first before the original bundle and then after it. Across all seven variants, I preserve each original question’s ID, instructions, criteria and option order:

{
	"type": "noul",
	"instructions": "Is two plus two equal to four?"
}

Question identity comes from its ID, so I match the department answer by that ID and its option names. I repeat identical requests to establish the baseline before attributing a difference to a changed sibling or order. The response reports a profile name, which can differ from the registry name requested in the body. I record both names and keep the requested model fixed across every variant.

Compare distributions before declaring an effect

I compare the whole distribution before looking at the winning department. A billing answer can stay billing while probability moves between technical and sales. We need all three named values to see that movement. I read the department answer by its question ID, then align its probabilities in the fixed billing, technical, sales order.

The function below compares two already captured response objects. I check the department fields it consumes and require matching response profile names. This is not a full response validator, and defining it produces no measurements.

import math


def compare(left, right):
    options = ("billing", "technical", "sales")

    def extract(response):
        profile = response["model"]
        if not isinstance(profile, str) or not profile:
            raise ValueError("Invalid model")
        answer = response["answers"]["department"]
        if answer["type"] != "choice":
            raise ValueError("Wrong answer type")
        probabilities = answer["probabilities"]
        if not isinstance(probabilities, dict) or set(probabilities) != set(options):
            raise ValueError("Wrong option keys")
        vector = tuple(probabilities[key] for key in options)
        if not all(type(value) in (int, float) and 0 <= value <= 1
                   and math.isfinite(value) for value in vector):
            raise ValueError("Probabilities must be finite numbers within [0, 1]")
        if abs(sum(vector) - 1) > 0.001 + 1e-12:
            raise ValueError("Probability sum outside tolerance")
        choice = answer["choice"]
        if not isinstance(choice, str) or choice not in options:
            raise ValueError("Invalid choice")
        if probabilities[choice] != max(vector):
            raise ValueError("Choice does not match displayed maximum")
        return profile, choice, vector

    model, left_choice, a = extract(left)
    right_model, right_choice, b = extract(right)
    if model != right_model:
        raise ValueError("Response model changed")
    differences = [abs(x - y) for x, y in zip(a, b)]
    delta_inf = max(differences)
    half_L1 = 0.5 * sum(differences)
    return {
        "delta_inf": delta_inf,
        "half_L1": half_L1,
        "choice": (left_choice, right_choice),
        "choice_changed": left_choice != right_choice,
        "within_tolerance": delta_inf <= 0.001 + 1e-12,
    }

For each pair, I return the largest absolute probability change and half the sum of absolute changes. Our proposed screening tolerance is 0.001, with 1e-12 arithmetic slack at the boundary. That threshold is our test choice, not a model guarantee. We retain both returned winners and flag changes even below the threshold.

The researched protocol makes three repetitions of seven variants, 21 sequential requests. I reverse submission order on the second repetition. We compare each focal singleton with every bundle, each perturbation with the original bundle, and repeated variants with their first valid response. These repetitions provide the baseline for interpreting bundle differences.

Ardana rounds returned probabilities to four decimal places, so I compare them without renormalizing. Equal vectors cannot establish identical internal logits. We keep the returned winner when displayed maxima tie, and half the sum of absolute changes describes the returned vectors without asserting exact normalization.

Before comparison, I exclude whole responses with HTTP, transport or JSON failures, including duplicate keys. Wrong IDs or types, missing fields, wrong probability keys or legends, and nonfinite or out-of-range probabilities also exclude a response. We reject inconsistent choices and probability sums outside the same 0.001 tolerance, with its arithmetic slack. Pairs with different response-model names are excluded too.

Keep failed bundles outside the probability comparison

I keep request failures separate from probability differences. A focal question can fit on its own while the larger bundle exceeds the row count or request token budget. An individual row can also exceed its context window. Those limits produce 413 errors, while invalid questions produce 422 errors. Neither error gives us a probability vector to compare.

Before I attribute a difference to bundling, I check what else changed. We need the same candidate options, model files, quantization, runtime and device across the run. A changed candidate set belongs in a separate experiment. I save the actual binary version and model profile alongside the results so we can inspect the setup later.

I have not run model inference for this post. Bundle differences, order effects and changed winners remain unmeasured. Our tolerance is a proposed screening threshold. Equal returned vectors would not establish identical internal logits, since we round probabilities to four decimal places.

Get the released Ardana v0.3.2 binary from the Ardana repository. Keep the configuration pinned and save it with your results so we can inspect the run together. Open the question planner on GitHub beside the test, then run the singleton and bundle variants against that setup and compare the results.