What missing information does to AI decisions

A question can require evidence the input lacks

Knowing that someone wants a refund does not tell us whether they qualify for one. Our existing Ardana ticket fixture asks whether the customer requests a refund. It does not ask whether the refund is authorized or whether a duplicate payment occurred. Those are different questions, even when they begin with the same ticket.

I would compare the question with the evidence we pass in. Ardana builds the input for its decision models from the state and questions we supply. If I ask about eligibility, I would first check that I supplied the records needed to establish it. Rewording the question does not put those records into the input.

Ardana derives its confidence and certainty measures from the answer probabilities. These describe the answer distribution; they cannot tell me which records are missing from the input. Before choosing a larger model, I would inspect what the current model could see.

An error floor depends on what is visible

Before I call an error unavoidable, I want to know what the model can observe. Hüllermeier and Waegeman’s 2021 review separates uncertainty in outcomes from uncertainty about the model and approximation uncertainty from limited training evidence. In their example, classes that overlap with one feature become separable with a second informative feature. Giving a model more training examples within the same representation changes a different part of the problem than adding an informative input. Extra dimensions can also make fitting harder.

An error floor depends on the evidence we make visible when asking the model to decide.

In the 12 October 2026 ProbabilityBench preprint, I can see that distinction in a controlled construction. Its three manually designed synthetic worlds start with hidden factors and deterministic outcomes, then expose selected attributes through fixed-template memos. Identical visible memos can correspond to different outcomes because the hidden states differ. Accounting for those hidden states gives exact conditional probabilities for each visible memo.

For binary classification, Bayes error is the lowest expected population misclassification rate achievable using those inputs. The corresponding Bayes accuracy bounds expected accuracy; a finite test set can score above it through label-sampling variation. The construction separately varies withheld attributes and an always-hidden residual factor, so missing signal and residual conditional uncertainty are separate choices. These are synthetic worlds with fixed templates and estimated population calibration integrals. I would use the construction to reason about missing evidence, without treating its floors as measured production error rates.

Disagreement does not tell you which label is wrong

I would review the labels before attributing every failed prediction to the model. Confident Learning studies errors in dataset labels, and its label-noise estimation builds on a class-conditional noise assumption. The work gives sufficient conditions for exact identification of label errors. Disagreement with a model alone does not prove that the recorded label is wrong.

Human disagreement deserves a separate check. ChaosNLI collected 100 annotations per example, totaling 464,500 annotations across SNLI, MNLI and αNLI examples. It found substantial disagreement on some language-inference examples and advocated examining collective label distributions. I take that as evidence that disagreement can reflect meaningful ambiguity. These measurements concern those inference datasets; they provide no measured uncertainty floor for our support decisions.

For a proposed label review, I would put each disputed example beside its source records and the annotation policy. Our existing ticket fixture asks whether the customer requests a refund. I would check that the reference label answers that question, rather than whether a refund is authorized. Then I would ask reviewers to explain which evidence and policy rule support their judgment.

When reviewers disagree, I would inspect their reasons before changing the label. A recorded mistake and an ambiguous instruction call for different corrections. If the policy leaves room for multiple readings, I would clarify it and review the affected labels again. That gives us a reviewed target for comparing what happens when we change the evidence supplied to the model.

Compare the evidence on matched cases

I would start with a held-out set of cases and review their labels against the source records and annotation policy. This is a proposed audit, with no measured result here. For each case, I would keep the question, instructions and model fixed across both runs. Asking about refund intent in one run and refund eligibility in another would change the task.

The first input would contain what the application currently supplies. For the second, I would add verified records that the application could legitimately have obtained before making the decision. I would record which evidence was added to each case. Where a record was unavailable, I would preserve that absence and keep the case in the comparison.

I would exclude final outcomes and notes created after the decision from both inputs. scikit-learn’s leakage guidance treats information unavailable at prediction time as leakage because it makes evaluation optimistic. We would also set aside the evaluation cases before choosing which information to add.

With reviewed labels, I would compare which decisions improved, regressed or stayed the same. I would keep unresolved label disagreements visible instead of silently turning them into definite errors. A gain would support adding those records for this model and question. An unchanged result would still leave open whether the records were useful or the model could use them.

The comparison would give us diagnostic evidence, not an exact Bayes error estimate. It cannot separate every source of remaining error or establish the best possible performance from those inputs. In Ardana, I would make the evidence change explicit in the caller-supplied state so we can inspect exactly what reached the model.

Make the missing record visible before another run

For this audit, I would start with the state our application sends to Ardana. The caller owns that state, and the decider path serializes it to build the questions the model receives. Adding another question does not retrieve a record we left out of the request.

Our existing ticket fixture asks whether a customer requests a refund. If I wanted to ask about eligibility instead, I would first inspect what evidence we supplied for that decision. Rewording the question cannot supply an omitted payment record or establish that a duplicate payment happened. The application has to obtain the relevant evidence and include it.

I would keep the answer distributions beside each pair of inputs during the proposed comparison. They show how the model distributes probability across the available answers. The confidence and certainty measures do not identify which record is missing or explain why the model is uncertain. To inspect the supplied evidence locally, I would start Ardana with:

ardana serve

Then I would open the playground at http://127.0.0.1:8000. My next step would be to put the question beside the supplied state. Before another run, I would check that the record needed to answer that question is actually present.