Rerank retrieved text with a local decision model

A reranker cannot recover a missing document

If retrieval misses the document that answers my query, a reranker cannot bring it back. It can move a useful passage up only if that passage is already in the candidate pool. In the monoT5 ranking definition, the output is an ordering of the retrieved candidates.

I keep retrieval, reranking, and generation separate for that reason. Retrieval searches the corpus and returns candidates. Reranking scores those candidates and changes their order; generation then produces an answer from the selected text. Nogueira and Cho’s BERT reranking paper describes these as three stages, each with a different job.

In their reranker, the model reads the query and each passage together. It predicts binary relevance independently for each passage and uses those probabilities to order the pool. Before generating an answer, I need that relevance judgment for every retrieved passage against the same query.

Ask the same relevance question for each candidate

I would give each retrieved passage the same query and ask the same relevance question. Does this passage help answer the query? Each judgment sees one query and one passage, with the relevance rubric held fixed across candidates. That is pointwise scoring: we score each candidate separately, then order the pool by those scores.

The BERT reranker predicts binary relevance independently for each passage and sorts by those probabilities. monoT5 normalizes the true and false token logits to obtain a relevance probability. Qwen’s reranker example reads the final-position yes and no logits and normalizes those two values. I am borrowing the pointwise structure here.

For Ardana, I would use a binary question that returns the probability of yes. We can supply descriptions of what counts as relevant and what counts as irrelevant. Ardana reads option-slot logits, applies a temperature-adjusted softmax, and formats probabilities directly. The proposed application is to repeat that relevance decision for every retrieved passage with the same query and rubric.

A choice among candidates asks a different question. Ardana normalizes a choice across the supplied options, while separate binary questions each normalize their own yes/no decision. Several passages can help answer the same query. I want a relevance score for each passage so we can sort them, rather than ask which single passage wins a choice.

I have not measured reranking quality or latency for this setup. Decider’s model card limits its calibration evidence to the evaluated datasets and probes, and warns about overconfidence on hard items. A probability of yes alone does not establish relevance calibration on our documents. I would compare this proposed application with a dedicated reranker such as Ettin before treating it as a replacement.

Send a query and document to the local API

I use the local decision API for this request. The example below follows the schema, but I have not run it. On macOS or Linux, install Ardana, download the model, and start the server with these commands. On Windows, I use the PowerShell install command below.

curl -fsSL https://ardana.ai/install.sh | sh
ardana pull decider:2b
ardana serve

irm https://ardana.ai/install.ps1 | iex

The server exposes both the API and the playground at the same local address. I send the query and one retrieved document together as JSON state. For a real request, replace the placeholders with your query and retrieved text, then save the body as request.json.

{
	"model": "decider:2b",
	"state": {
		"query": "<query>",
		"document": {
			"id": "doc-1",
			"text": "<retrieved text>"
		}
	},
	"questions": {
		"relevance": {
			"type": "noul",
			"instructions": "Is this document relevant to the query?",
			"criteria": {
				"true": "The document helps answer the query.",
				"false": "The document is unrelated or only shares keywords."
			}
		}
	}
}

I chose the payload’s inner names to carry the query, document text, and document ID together. Those names belong to the caller, as does the name of the relevance question. The API requires an input state and a question collection, but model selection is optional. The criteria spell out what yes and no mean for this decision.

We send the saved body to the local decision route with the command below. The released API exposes this endpoint but has no separate reranking route. We sort the candidates in application code. I can also pass the saved body through the CLI with the alternative below.

curl http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d @request.json

ardana run decider:2b --request request.json --json

I read the probability of yes directly from the named answer. Its numeric value is the relevance score for this query and document. This is the response lookup for the question above:

response["answers"]["relevance"]["noul"]

The answer includes its type and the numeric probability, with no additional confidence field. I also get the resolved model profile and token usage in the response. Usage reports input tokens and sets output tokens to zero. We can now attach the score to the original document before doing any sorting.

Keep document identity through sorting and failures

For this proposed loop, I send the same query and relevance rubric with each candidate document. A pool of K candidates needs K API decisions. I keep the original ID and full document beside each returned score in caller code. Sorting changes their position while keeping the complete document record together for generation.

Sorting also belongs in our application. I build the scored records in retrieval order, using the returned relevance value as the sort key. Python’s descending sort is stable, so equal scores keep that input order:

sorted(scored, key=lambda row: row["relevance"], reverse=True)

I use retrieval order as the tie policy because Ardana rounds returned relevance probabilities to four decimal places. Equal returned values therefore have a predictable order without another comparison. We select the generation context only after sorting, keeping each selected document attached to its ID and full text.

I handle request failures separately in this loop. HTTP 422 means an invalid request, and HTTP 413 means a capacity violation. I keep each failure attached to the candidate ID and handle it before selecting context. A rejected request gives us no relevance score. Assigning it a low score would turn an API problem into a judgment about the document.

We also need to account for input limits before sending the document. The default planner limits are 1024 scoring rows, 36864 tokens per row, and 1048576 total row tokens per request. The model’s context window also applies. Ardana refuses oversized state rather than silently truncating it, so I cannot assume that a long document will be shortened for this request.

Compare rankings on the actual retrieved pool

With sorting in place, I would compare three orders on a small set of queries with local relevance labels. Keep the retrieved candidates fixed and compare their initial order, Ardana’s order and a specialist reranker’s order. The Ettin rerankers provide a dedicated cross-encoder baseline. This checks whether our relevance decisions improve the order of documents retrieval actually found.

Sentence Transformers’ reranking evaluator reports MRR@k, NDCG@k and MAP and supports comparison against the supplied original ranking. I would supply the retrieved documents in their original order and disable its default insertion of all positive documents with the setting below. A missing relevant document must stay missing in this comparison, because reranking cannot recover it.

always_rerank_positives=False

The scores need local checks too. Decider’s calibration evidence covers its evaluated datasets and probes, and hard items can produce overconfident scores. I would inspect disagreements against our labels before treating a high relevance score as dependable.

I have not run this comparison or measured any ranking-quality or latency gain. I would record ranking metrics and elapsed time on the same candidate pools, then inspect which documents moved into the context selected for generation. Grab Ardana and try the request on your existing retrieved candidates.