Select few-shot examples for local decision models

An extra example is another change to test

When I add a labeled example to a routing prompt, I change the input used to make the decision. I would test that change. This is in-context learning: examples guide a prediction without updating the model’s weights.

Selection and order deserve separate tests.

Choosing semantically similar examples improved on random selection in the GPT-3 tasks studied by Liu et al.. For ordering, Lu et al. found substantial sensitivity across 24 orders of four demonstrations in GPT-2/GPT-3 text classification. I would use those findings to motivate an Ardana experiment, without assuming their results transfer.

Here, demonstration order means the sequence of labeled examples in the prompt. Answer-option order means the sequence of labels available for the new prediction. I would keep those options in place while rearranging the examples.

My selection hypothesis starts with tickets that share invoice and export vocabulary but need different routing labels. I want to test whether those examples help the model use the routing rule. The prepared requests passed input validation, but no inference was run. There is no measured improvement or best order yet.

Put labeled references in ordinary state

I put labeled reference tickets in the same state as the ticket I want to route. The demonstrations array is my own convention inside ordinary JSON state. Ardana serializes that state into context; there is no native few-shot field or dedicated demonstration parser. I save my example as request.json, with synthetic tickets and labels I assigned myself.

{
	"model": "decider:2b-q4_k_m",
	"state": {
		"demonstrations": [
			{
				"ticket": "Export works, but I was billed for seats I removed.",
				"department": "billing"
			},
			{
				"ticket": "The invoice PDF download button throws an error.",
				"department": "technical"
			},
			{
				"ticket": "Does Enterprise include invoice exports for procurement?",
				"department": "sales"
			}
		],
		"current_ticket": "The billing page freezes when I click Download invoice."
	},
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Route only current_ticket. demonstrations contains labeled reference tickets. Use billing for charges or refunds, technical for broken functionality, and sales for purchasing or plan-feature questions.",
			"criteria": ["billing", "technical", "sales"]
		}
	}
}

I route tickets by the problem they describe: charges and refunds go to billing, broken functionality goes to technical, and purchasing or plan-feature questions go to sales. The selected references share invoice or export vocabulary across different labels. I want to test whether these references help the decision follow the rubric when vocabulary alone would be ambiguous.

For the current ticket, I assign technical as the expected answer because the page freezes during a download. The mention of billing does not change that rule. I keep this expected answer outside the request, in the evaluation record. The reference labels belong in the input because they are examples; the current ticket’s label is what I will compare against the returned choice.

I pull the explicitly tagged model below before running the request. The JSON and the command use the same model tag. The positional model selects CLI execution, so I keep those two values identical.

ardana pull decider:2b-q4_k_m

The run command below reads the complete saved request and writes the response to a file. The request has passed the released request and rendering checks. I have no inference result for it yet, so there is no observed choice or probability to report.

ardana run decider:2b-q4_k_m --request request.json --json > response.json

Change selection before changing order

I would start with three conditions: no demonstrations, general examples, and the selected invoice/export examples in their original order. The empty condition removes only the demonstrations; I keep the routing instructions. General examples cover a duplicate charge, an app crash, and a quote request. The selected set puts overlapping vocabulary on different sides of the routing boundary, with one example per label. That is my selection hypothesis, with no measured gain behind it.

For that first comparison, I keep both example sets in billing, technical, sales order. Then I would reorder the selected set to test order separately. Calling its examples s1, s2, and s3, the six orders are 123, 132, 213, 231, 312, and 321. Together with the empty and general conditions, that makes eight conditions, or 8*N requests for N fixed evaluation items before retries.

I use the saved request.json as the starting point for this generator. It writes all eight condition files for the current ticket, with the original selected order included among the permutations. Every file keeps the model and quant, question ID, rubric, answer-option order, and current ticket fixed. I use the same JSON serialization for every condition, changing only the demonstration contents or sequence.

import json
from copy import deepcopy
from itertools import permutations
from pathlib import Path

base = json.loads(Path("request.json").read_text())
selected = base["state"]["demonstrations"]
general = [
    {"ticket": "My card was charged twice. Please refund the duplicate.",
     "department": "billing"},
    {"ticket": "The app crashes before login.", "department": "technical"},
    {"ticket": "Please send a quote for the team plan.", "department": "sales"},
]
conditions = {"zero": [], "general": general}
for order in permutations(range(3)):
    name = "selected-" + "".join(str(i + 1) for i in order)
    conditions[name] = [selected[i] for i in order]

output = Path("conditions")
output.mkdir(exist_ok=True)
for name, demonstrations in conditions.items():
    request = deepcopy(base)
    request["state"]["demonstrations"] = demonstrations
    (output / f"{name}.json").write_text(json.dumps(request, indent=2) + "\n")

These are generated inputs; I have no inference results from this comparison. The research checks passed request validation and rendering, but an actual client run can still fail. I have no automatic retriever here, and no evidence that one selected order wins. I would run the same fixed cases through each condition before judging whether the selected examples help.

Choose on development cases and freeze the test

I would freeze three disjoint partitions: a demonstration pool, development cases, and a final test. Duplicate or related tickets belong in the same partition, so a reference ticket cannot leak the answer to a related test case. Demonstration labels belong in the input. The current case’s gold label stays in the evaluation record, and final-test labels never enter prompts or example selection.

Before running the comparison, I would declare its conditions and keep them fixed. I would choose the deployment configuration using development results, lock it before final testing, and report the predeclared conditions on the final test. Choosing a winner from those test results would make the test another tuning step, so I would not present that winner’s score as an unbiased estimate.

For accuracy, I would divide correct decisions by all evaluation cases, counting failed requests as incorrect. For each label, recall is correct decisions on cases with that gold label divided by all cases with that gold label. Failed requests remain in those denominators and contribute no correct decision. If a label has no cases, I would leave its recall undefined. I would report failure counts separately too.

To measure sensitivity between two conditions, I would divide different choices by the number of paired cases where both requests succeeded. I would report failures alongside this flip rate so excluded cases stay visible. A stable choice can still be wrong, and a flip does not establish an accuracy improvement. Our verification checked request construction without running inference, so there is no measured best order here.

Run the same cases with the chosen examples

I would run the locked configuration and the baseline on the same held-out tickets, saving the evaluation ID, condition, ordered example IDs, exact request, and raw response for each run. The expected label stays outside the request. I would record the exact resolved model identity and quantization, then check that the saved request and command select the same model.

ardana run decider:2b-q4_k_m --request request.json --json > response.json

Before the next run, I would preserve both files under that evaluation ID and condition. I would compare accuracy, count failed requests as incorrect, and report those failures separately. Where both runs succeed, I would count changed choices to see whether the examples affected the decision. The expected label tells me whether a change helped.

The requests here passed the released request and rendering validation, but no inference was run. I have no measured probabilities, accuracy gain, or best example order to recommend. I would choose the configuration on development cases and lock it before this final comparison. The Ardana repository has the installation and API examples; start there, then run your saved baseline and chosen demonstration requests against the same cases.