Compare local decision models with text classifiers

A fixed label task deserves a supervised baseline

I want a support router to put each incoming message into a department from a fixed list. Our application needs a label it can act on. If we can label examples of that task, a supervised text classifier belongs in the comparison before we choose what to deploy.

Ardana reads state and typed questions locally and returns probabilities for named options without generating text. For a supervised baseline, I can turn messages into TF-IDF features and fit logistic regression. I also want a dummy classifier that ignores the text and predicts the training majority class. That tells us what we get from class frequency alone.

I have no measured winner to report for this task. We need the same task definition, source evidence, and permitted label pools for each method. Prompt edits spend adaptation labels too, so I would keep a development-tuned prompt separate from the frozen zero-shot reference. That reference consumes no target adaptation labels. Equal access to target labels still does not give our methods equal pretrained knowledge.

Give every method the same information budget

I count adaptation labels before comparing methods. We share the task definitions, source evidence, and permitted label pools across conditions. Any labels used for prompt edits, tuning, demonstrations, or calibration come out of that adaptation budget. I report evaluation labels and annotation labor separately.

Here is the schedule I propose, with no run behind it yet. At 30 total adaptation labels, I reserve 24 for fitting and 6 for development; at 120, 96 and 24; at 480, 384 and 96. We repeat these subsets with seeds 11, 23, and 47 to see how much the selected examples change the development curve. I have not tested whether these allocations are sufficient.

For the frozen zero-shot reference, I count zero target adaptation labels. Once I change its prompt using development labels, I record a separate condition and charge those labels to it. Equal access to target labels still leaves prior pretraining unequal. Our comparison needs that distinction written down.

Before fitting preprocessing, I split the data and keep dependency groups apart. Grouped stratification can try to balance classes, but it cannot guarantee that balance. We learn the vocabulary and IDF within each permitted fit subset, then transform development and test text with that fitted vectorizer. I choose a method and budget using repeated development subsets. The final holdout stays untouched until we freeze every condition.

Separate lexical and frozen semantic features

I would start with DummyClassifier using its prior strategy. It ignores text, predicts the training majority class, and returns the training class frequencies as probabilities. That gives us a reference for what the label distribution alone provides.

For lexical features, I would pair TfidfVectorizer with LogisticRegression. The vectorizer can include unigrams and bigrams; its default word tokenizer ignores punctuation and one-character tokens. Logistic regression accepts sparse or dense features and uses regularization by default. We split first, then fit vocabulary and IDF only on each permitted training subset. Validation and test text pass through that fitted vectorizer without updating it.

Related examples need to stay together in our splits. GroupKFold keeps groups apart; StratifiedGroupKFold also attempts class balance. We still need to inspect the resulting splits, because class balance is not always achievable.

The semantic condition would use sentence-transformers/all-MiniLM-L6-v2 to produce 384-dimensional vectors. I would leave the encoder unchanged and fit only logistic regression on those vectors. SetFit fine-tunes Sentence Transformers, so I would treat it as a separate adaptation condition. Calling that a frozen baseline would hide that its encoder learned from our labels.

I would set an explicit shared input and length policy for every method. This encoder defaults to truncating beyond 256 word pieces. We need to record what text survives, because equal label access does not fix unequal source evidence.

Finally, I would map classifier probability columns back to their class names before comparing them with Ardana’s label-keyed probabilities. The same task definitions and permitted label pools apply across these conditions. Any tuning spends our adaptation-label budget.

Send the same task to the local decision model

I give Ardana the same ticket evidence and department definitions as the supervised conditions. For the frozen zero-shot reference, I fix that rubric without spending target adaptation labels. If I edit it after inspecting labeled development examples, I count that work against the adaptation budget and report a separate condition.

For this condition, I use the catalog’s 1.9B-parameter model. Its default quantized GGUF is 1274396800 bytes. I record that as the weight artifact size, separate from resident memory, which the download size does not measure.

On macOS or Linux, I install v0.3.2 with the command below. Next I pull the model weights. Then I start the server and leave that terminal running.

curl --proto '=https' --tlsv1.2 -LsSf https://github.com/ardana-ai/ardana/releases/download/v0.3.2/ardana-installer.sh | sh
ardana pull decider:2b-q4_k_m
ardana serve

From another terminal, I send the complete request below. Ardana reads the state and typed question locally, returning named option probabilities without generating text. This request pairs the shipped duplicate-charge ticket with our authored routing rubric. We have no observed answer for it.

curl -fsS http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{"model":"decider:2b-q4_k_m","state":"My card was charged twice for order A-104. Please refund the duplicate.","questions":{"department":{"type":"choice","instructions":"Which department should handle this ticket?","criteria":{"billing":"Charges and refunds","technical":"Broken functionality","sales":"Purchasing and plan questions"}}}}'

I use these answer paths to read the chosen department and named probability dictionary. Our evaluator uses logistic regression’s probability method and class-order attribute, shown below, to map its probability columns to those same label names.

answers.department.choice
answers.department.probabilities
predict_proba
classes_

Choice confidence rescales the largest probability using the number of options. In the formula below, I use n for the option count and p_max for the largest probability. Public probabilities are rounded to four decimal places, and the response exposes no raw logits. Our comparison therefore uses the serialized probabilities at that precision.

confidence
(n * p_max - 1) / (n - 1)

Choose from frozen test evidence

I would select the method and label budget on development curves, then freeze every condition before opening the final holdout. My report would show macro-F1, per-class precision and recall, and class counts, with failed cases kept visible. I would separate training-subset variability from paired test uncertainty, resampling dependent groups together.

I would report log loss and Brier score in separate columns to measure probability quality. Returning probabilities does not guarantee calibration for either approach on our task. Measurements would cover the full deployed path, including feature extraction or embeddings. I would report batch and individual prediction separately, and measure resident memory separately from weight-file size.

Our deployment choice would need to meet the classwise quality requirements on the frozen test and fit our latency and memory limits. Taxonomy maintenance belongs in that decision too: changing request-time option definitions differs from changing a fitted classifier’s target classes. Either change needs fresh task evaluation. Try Ardana on GitHub for your own comparison, and tell me how you choose.