Blog

Build weak labels for local decision models

Create candidate labels from fallible rules with explicit abstention and conflicts. Inspect vote coverage and keep weak labels separate from evaluation.

A small brown groundhog watches color tokens collect in matching, mixed and empty compartments.

Agreement between rules is not ground truth

I can send a support ticket to the wrong department even when every rule agrees. Suppose my rules all treat the same phrase as evidence for the wrong destination. The resulting label looks settled because nothing disagrees with it. That is the false agreement I need to catch before I trust a label.

Weak supervision gives me provisional labels built from fallible labeling functions. These are pieces of code that propose a class or abstain. Abstention means the function supplies no vote. Generating weak labels is a separate step from training a model on them, as the Snorkel paper describes.

Before accepting a candidate, I can measure coverage and conflict without an answer key. Measuring how often a labeling function is correct requires gold labels, as Snorkel’s analysis documentation explains. I want the outputs to preserve each function’s vote or abstention so the next step does not treat the rules as more certain than they are.

Give every rule a label or an abstention

I give every rule the same contract: return a class ID or abstain. In this example, billing is zero, technical is one, and minus one means no vote. This follows Snorkel’s label-matrix convention, where class IDs start at zero and abstention sits outside the classes. When one of these rules finds no keyword match, it abstains instead of voting against that department.

I kept the rules in our hypothetical labeling fixture deliberately simple so we can see exactly what triggers each vote. Each function matches one keyword without regard to case, and each candidate record preserves votes under the original function names.

import re

ABSTAIN, BILLING, TECHNICAL = -1, 0, 1


def lf_refund(text):
    return BILLING if re.search(r"\brefund\b", text, re.I) else ABSTAIN


def lf_charge(text):
    return BILLING if re.search(r"\bcharge\b", text, re.I) else ABSTAIN


def lf_crash(text):
    return TECHNICAL if re.search(r"\bcrashes\b", text, re.I) else ABSTAIN


LFS = (lf_refund, lf_charge, lf_crash)

Our hypothetical ticket says, “No refund or charge issue: the app crashes.” Both billing rules fire on words inside a negated issue, while the crash rule votes technical. The authored rubric expects technical here, so I have deliberately constructed a failure we can inspect. The functions recognize keywords; they do not understand which issue the ticket denies.

For each complete candidate record, I retain the ticket ID, rule revision, and named votes alongside the provisional label, status, overlap, and conflict. Those names let me trace a billing vote back to the refund or charge rule. The revision records which rule set produced it.

Inspect conflicts before combining votes

I inspect the votes before accepting a candidate. Coverage counts rows with at least one vote. Overlap counts rows with at least two votes, even when they agree. Conflict counts rows where non-abstaining votes disagree, and all three dataset rates use every row as the denominator.

Our first hypothetical vector produces a billing candidate. The majority wins, but I keep the conflict in the record because another rule voted differently.

[0, 0, 1]

The second hypothetical vector has overlap, conflict and a tie. I withhold the candidate because neither class wins. An all-abstain row also gets no candidate, but I keep it separate because no rule supplied a vote.

[0, -1, 1]

Across our six authored rows, five have coverage, four have overlap and three have conflict. We get four candidate labels, one tie and one uncovered row. These counts describe the hypothetical fixture, so I use them to check the aggregation behavior.

Copying the refund rule changes our hypothetical T04 from a tie to billing. I have changed no ticket text. The duplicate gets another vote because this baseline gives every non-abstaining rule equal weight. More votes can come from repeating the same evidence, so I inspect which rules supplied them before trusting the winner.

Snorkel v0.9.7 also gives each non-abstaining rule one vote. Its probability method assigns mass to winning classes rather than returning raw vote fractions. When I use that library, I explicitly select its supported abstention policy for ties. We can calculate coverage and conflict without an answer key, but measuring rule accuracy still needs gold labels.

Keep weak targets separate from independent evaluation

I treat a learned label model as another set of assumptions to inspect. Snorkel v0.9.7 estimates rule-output probabilities conditional on an unobserved class and assumes the rules are independent given that class. Related regexes and overlapping knowledge sources can create dependencies. The Snorkel paper discusses dependency modeling separately from the conditionally independent implementation.

We also choose a class prior, either by supplying class balance or estimating it from development labels. A learned weak-label probability remains provisional. I call probabilities calibrated when they agree with observed outcome frequencies, which neither vote fractions nor learned weak-label probabilities establish on their own.

For evaluation, I keep provisional targets, development review, and held-out gold labels separate. Development review can inform rule changes and prior estimates. Held-out gold labels stay out of those choices and provide an independent answer key. Comparing a model with our provisional targets measures agreement with the rules that produced them. Our example’s review labels are authored fixtures, not independently reviewed gold.

Before the local comparison, I keep rule votes, candidate labels, and reviewed answers in separate evaluator records. Ardana renders request state into model input, so those values would expose the answer or its derivation. Our request builder passes only ticket text through. Weak labeling remains our application code; Ardana supplies the inference step.

Compare against a local model with clean state

I keep the model comparison separate from candidate generation. Ardana renders the request’s state into model input, so rule votes and reviewed answers belong in separate evaluator records. For this hypothetical ticket, I pass only the original text. The request supplies department descriptions and a rule for negation and competing active issues.

Save this complete illustrative request as request.json. Its ticket is an authored example, not an independently reviewed test case. I checked the request structure, but I have not sent it to a model, so there is no prediction to report.

{
	"model": "decider:2b",
	"state": {
		"ticket": "No refund or charge issue: the app crashes."
	},
	"questions": {
		"department": {
			"type": "choice",
			"instructions": "Which team handles the active issue? Ignore explicitly negated issues. If both remain active, choose billing.",
			"criteria": {
				"billing": "Active refund or charge issue.",
				"technical": "Active application failure."
			}
		}
	}
}

I install Ardana on macOS or Linux, pull the model, and run the saved request through the documented CLI. This is the inference step; weak labeling remains application code. The JSON response includes the chosen department and its probabilities.

curl -fsSL https://ardana.ai/install.sh | sh
ardana pull decider:2b
ardana run decider:2b --request request.json --json

For an HTTP comparison, I start the local server, then send the same saved request from another terminal.

ardana serve
curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

My next step is a model run against independently reviewed examples, with the results kept in separate evaluator records. This authored ticket gives us a request to try, not an accuracy result. Grab Ardana on GitHub and run the saved request.