How to test whether a model follows changing rules

A changed answer is not enough

I want a refund decision to change when the policy changes. In this hypothetical test, the purchase is 14 days old. My complete policy is to approve purchases at most 30 days old and deny the rest. Then I change only the limit to 7 days. I assign the expected labels: approve for the first policy, deny for the second.

A changed answer still tells me very little. Suppose a model predicts deny for the first policy, then approve for the second. That hypothetical model flips its answer and gets both wrong. Those are expectations I wrote and predictions I imagined. I have not run the fixtures through a model.

In the option-order post, we tested what happens when the choices move. Now I change the rule so the same purchase requires a different answer. Each answer has to match the policy in its request. I write down that expected relationship before running the pair.

Give every edit an expected relationship

I use CheckList’s test types to decide what each case should check. Minimum functionality tests use small examples with known labels to isolate a behavior. Invariance tests change the input while keeping the intended label fixed. Directional tests specify an expected relationship after an edit: a target label or a probability moving in a specified direction, which need not change the selected label.

For label-changing pairs, I use Gardner and colleagues’ contrast sets. These make small, meaningful edits that typically change the correct label. Their contrast consistency metric requires every example, including the original, to be correct.

I give each edit an expected relationship before running it: what changes, what stays fixed, and which answers count as correct.

With our invented refund policy, I propose separate rule-only and fact-only tests. In the rule-only pair, the purchase stays 14 days old while the refund window shrinks from 30 days to 7. I derive the expected labels from the policy: approve for the original, deny for the changed rule. For the fact-only pair, I keep the 30-day window and change the purchase age from 14 days to 31. The expected labels are again approve and deny.

The order control asks for a different relationship. I reverse the options while keeping the original policy and purchase age, so the expected label stays approve. Together, these cases let us check whether a changed rule, a changed fact, or only the option order affects the answer as intended.

Keep the policy pair small enough to inspect

I keep this example to one purchase age and one complete, invented refund policy. The baseline is rule-a.json, with an intended answer of approve. These fixtures have not been sent to a model. I assigned the expected labels from the test policy; they are not observed answers.

{
	"model": "decider:2b",
	"state": { "days_since_purchase": 14 },
	"questions": {
		"refund": {
			"type": "choice",
			"instructions": "Apply this complete test policy: approve a refund if days_since_purchase is at most 30; otherwise deny it.",
			"criteria": ["approve", "deny"]
		}
	}
}

The purchase is 14 days old, inside the baseline’s 30-day window. I keep the expected labels in a separate expected-labels.json file, outside the request’s state. That leaves the request with the purchase fact, the policy to apply, and the available choices. We can inspect the intended answer before we ask a model to choose.

For rule-b.json, I replace only 30 with 7 in the instruction text. The purchase stays 14 days old, and the intended answer becomes deny. For fact-b.json, I start from the baseline again and change only the purchase age from 14 to 31. The 30-day policy stays fixed, so I expect deny there too.

I start from the baseline again for order-control.json and reverse only the choices, putting deny before approve. The purchase age and policy stay fixed. I still expect approve, because moving a choice does not change this policy. Each variant gives us one edit to inspect against the original, with its expected label recorded separately.

The CLI help shows this general form for sending a whole request file. Here, request.json is the help example’s filename. I would use the specific fixture filenames and explicit model size for this test.

ardana run decider --request request.json --json

To run the policy pair, I would use these commands from the directory containing the saved request files. They select decider:2b and send each complete request separately. The model name identifies what we want to test; it does not tell us whether either answer will be correct. Once we have responses, I would compare the selected choice names with the expected labels.

ardana run decider:2b --request rule-a.json --json

ardana run decider:2b --request rule-b.json --json

Score the whole pair

I use Gardner and colleagues’ contrast consistency definition: every example in the set must be correct, including the original. For our rule-change pair, that means checking both expected labels together. The invented policy calls for approve on the original request and deny after the rule changes. I count the pair as passing only when both answers match.

Consider two hypothetical prediction patterns, not results from Ardana. If a model always predicts approve, it gets one of two answers right and fails the pair. If it predicts deny and then approve, it changes its answer but gets both wrong. I would keep individual-answer accuracy alongside the pair result so a partial success cannot hide a failed contrast.

Named choices are my primary check here. I compare the selected names with the expected names, including when I reverse the option order. I would retain the probability maps as diagnostics when inspecting a failed pair. A probability movement does not rescue an incorrect named choice under this test.

What I can score also depends on the answer type. I would declare the decision threshold for a yes-probability answer before running the fixture, and the acceptance rule for an expected-level score before judging the answer. Those rules belong with the test definition so I can apply the same judgment on every run. We have not sent these fixtures to a model, so these are proposed checks with authored labels, not measured accuracy.

Expand the suite without changing what it proves

Yuichi Tateno’s S1MB includes related synthetic tests. Contextual Noul changes state and criteria together, so I cannot isolate rule effects. GPT-6-Astra generated and checked questions and intended answers without independent human validation. Training overlap remains possible; Tateno cautions that these simple tasks do not establish unrestricted generalization.

For our next cases, I want a domain expert to write the edits without consulting the model’s answers. I would ask another person to derive the expected labels independently, then resolve disagreements before running anything. I would start with focused regression cases for negation, time boundaries, and policy exceptions, using templates to vary the examples. Each case should be attached to the behavior it tests, so a failure points to something we can inspect. As the contrast-set paper cautions, this is evidence about local behavior around selected examples.

Before adding a proposed case, I would save the reason for its expected answer beside the fixture. A reviewer should be able to derive that answer from the test policy alone. If we disagree, I would revise the case before adding it to the regression suite.

The supplied fixtures have not been sent to a model, so we have no measured result for them. My next step is to commit the requests alongside the separate expected-label file. From that directory, try the pair:

ardana run decider:2b --request rule-a.json --json
ardana run decider:2b --request rule-b.json --json