Blog

Evaluate complete agent decision workflows

Score complete agent runs by final state and required replies. Track recovery and stopping reasons, then distinguish repeated success from a lucky attempt.

A small brown groundhog follows stepping stones toward a wooden tray, with a short detour around one broken stone.

A correct next action does not finish the task

In the refund example I wrote, the final reply claims success, but the stored state is unchanged. The requested refund is 2500 cents. I count the run as a failure. This is a scoring fixture, not a measured model result.

That gap is why I care about the workflow experiment in JEVal v2. Jev selects an available tool or chooses to respond, while a separate DeepSeek-V4-Flash model generates the arguments or response text. With the tools and environment held fixed, episode success fell from 77.6% for the base agent to 66.1% for Jev plus the language model. These are the paper’s results, not an Ardana benchmark.

The authors trace failures to unsuitable action choices and incorrect arguments, followed by repeated lookups, retries, and some timeouts. Each task ran once per system, so we cannot read those scores as repeated-trial reliability. I want to keep that distinction visible when wiring a decision model into an agent. Choosing the next action is one part of a loop whose later state still needs checking.

Keep the decision model inside a visible loop

I keep Ardana inside an application-owned loop. It returns a typed decision, and my application owns the executor, state and history. The shipped setup commands pull the decision model and start the local server:

ardana pull decider:2b
ardana serve

The pinned API definition accepts JSON state and a map of questions. I use that shape for the authored request below, saved as example/request.json. I have not sent this request or run model inference for it.

{
	"model": "decider:2b",
	"state": {
		"request": "Refund the duplicate 2500-cent charge on order A-104.",
		"observation": {
			"order_id": "A-104",
			"last_tool_result": "Temporary lookup failure"
		},
		"history": [
			{
				"action": "lookup",
				"result": "error"
			}
		]
	},
	"questions": {
		"next_action": {
			"type": "choice",
			"instructions": "Choose the next action using the observed history. A failed lookup has not established that a refund was completed.",
			"criteria": {
				"lookup": "Read the current order and payment records.",
				"refund": "Issue the duplicate-charge refund after the necessary records have been read.",
				"respond": "Report the observed outcome to the user."
			}
		}
	}
}

My application defines the choices lookup, refund and respond. They are not built-in Ardana tools. Here I include the failed lookup in the history and explicitly tell the selector that it has not established a completed refund. A future choice response would expose the selected name at this path:

answers["next_action"]["choice"]

From there, my application must supply the arguments and execute the selected operation. It must record the result, update the visible state and history, then make a new decision request. Ardana does not generate those tool arguments or run the evaluation harness. I need that boundary visible in the trace so I can inspect the choice, the supplied arguments and the operation’s result separately.

For each proposed trial, I would preserve the state visible before the action alongside the raw decision. That gives me a record of what the selector could see. I would also retain the final state, user-facing reply and stopping reason so the later scorer can judge the completed run.

Score the terminal state and required reply

I score the state left behind when the run stops and the information the user received. A refund reply cannot compensate for money that was never refunded. In tau-bench, the reward combines the final database check with required information in the agent’s replies. That checker can still miss policy violations, so I treat these checks as a defined success condition rather than proof that every rule was followed.

My complete offline scorer uses only the Python standard library. Save it as score_traces.py. I wrote all six traces to show successful completion, recovery, false completion, a wrong address, and a turn limit. These are authored fixtures, with no model calls or measured Ardana results.

"""Authored offline fixtures. No model inference or benchmark measurements."""
from fractions import Fraction
from math import comb

GOALS = {
    "refund": {"refund_cents": 2500, "status": "refunded"},
    "address": {"address": "42 Pine St", "status": "updated"},
}
REQUIRED = {"refund": "2500 cents", "address": "42 Pine St"}

# Each event is (action, result). These are invented observations, not API answers.
TRACES = [
    {"task": "refund", "trial": 1, "events": [("lookup", "ok"), ("refund", "ok"), ("respond", "ok")], "final": {"refund_cents": 2500, "status": "refunded"}, "reply": "Refunded 2500 cents.", "stop": "finished"},
    {"task": "refund", "trial": 2, "events": [("lookup", "error"), ("lookup", "ok"), ("refund", "ok"), ("respond", "ok")], "final": {"refund_cents": 2500, "status": "refunded"}, "reply": "Refunded 2500 cents.", "stop": "finished"},
    {"task": "refund", "trial": 3, "events": [("lookup", "ok"), ("respond", "ok")], "final": {"refund_cents": 0, "status": "paid"}, "reply": "Refunded 2500 cents.", "stop": "finished"},
    {"task": "address", "trial": 1, "events": [("lookup", "ok"), ("update", "ok"), ("respond", "ok")], "final": {"address": "42 Pine St", "status": "updated"}, "reply": "Address updated to 42 Pine St.", "stop": "finished"},
    {"task": "address", "trial": 2, "events": [("lookup", "error"), ("lookup", "error"), ("respond", "ok")], "final": {"address": "9 Oak St", "status": "unchanged"}, "reply": "Could not complete the update.", "stop": "turn_limit"},
    {"task": "address", "trial": 3, "events": [("lookup", "ok"), ("update", "ok"), ("respond", "ok")], "final": {"address": "24 Pine St", "status": "updated"}, "reply": "Address updated to 24 Pine St.", "stop": "finished"},
]


def success(trace):
    return (trace["stop"] == "finished"
            and trace["final"] == GOALS[trace["task"]]
            and REQUIRED[trace["task"]] in trace["reply"])


def pass_all(k):
    scores = []
    for task in GOALS:
        trials = [trace for trace in TRACES if trace["task"] == task]
        n = len(trials)
        c = sum(success(trace) for trace in trials)
        if not 1 <= k <= n:
            raise ValueError("k must be between 1 and trials per task")
        scores.append(Fraction(comb(c, k), comb(n, k)))
    return sum(scores, Fraction()) / len(scores)


error_episodes = [trace for trace in TRACES
                  if any(result == "error" for action, result in trace["events"])]
recovered = sum(success(trace) for trace in error_episodes)

# These assertions verify the author's arithmetic, not a model's quality.
assert [success(trace) for trace in TRACES] == [True, True, False, True, False, False]
assert pass_all(1) == Fraction(1, 2)
assert pass_all(2) == Fraction(1, 6)
assert (recovered, len(error_episodes)) == (1, 2)
assert sum(trace["stop"] == "turn_limit" for trace in TRACES) == 1
print("Authored fixtures only; no model inference")
print("successful episodes: 3/6")
print("pass^1: 0.5000")
print("pass^2: 0.1667")
print("error episodes recovered: 1/2")
print("turn-limit episodes: 1/6")

Our miniature database is the entire final dictionary for each task. I require it to equal the task’s goal, require the reply to contain the expected information, and require the run to have finished. The refund trace with an unchanged paid state fails even though its reply claims success. The address trace that updates the wrong destination fails too.

I keep the stopping reason separate from the reply so a turn limit remains visible. The recovered refund takes an extra lookup after an error and still passes. We accept that route because the scorer does not compare action sequences against one fixed path. Different conversations and paths can reach the same expected state.

The assertions check my fixtures and arithmetic. They do not establish model reliability or show that retrying caused recovery. I also keep trials grouped by task to calculate how often the whole task succeeds across repeated attempts.

Measure repeated success on the same tasks

I want to know whether we can finish the same task repeatedly. Tau-bench defines pass^k as all k independent trials of the same task succeeding, averaged across tasks. Pass@k asks whether at least one succeeds. I can get one successful attempt without getting consistent success across attempts.

The estimator in our scorer counts the ways to choose k successes from c, then divides by the ways to choose k trials from n. We compute that combination ratio separately for each task before averaging. At k = 2, only pairs where both trials succeed contribute. I run the saved scorer with:

python3 score_traces.py
Authored fixtures only; no model inference
successful episodes: 3/6
pass^1: 0.5000
pass^2: 0.1667
error episodes recovered: 1/2
turn-limit episodes: 1/6

Our authored tasks have 2/3 and 1/3 successes, giving 3/6 successful episodes overall. Their average gives pass^1 = 0.5000. For pairs, the first task contributes 1/3 and the second contributes 0, so pass^2 = 1/6, displayed as 0.1667. I read this drop as deterministic arithmetic from the fixtures; we have not sampled independent agent runs.

One of the two error-containing episodes succeeds after another lookup, which gives the recovery count of 1/2. We also have one turn-limit stop among six episodes. I wrote those outcomes into the example, so neither count estimates real recovery behavior or any benefit from retrying. Replaying these fixed traces cannot show how a changed choice would alter later states.

Run each policy through its own trajectory

For a live comparison, I would reset the sandbox before every trial and keep the underlying task fixed. I would record task and trial identifiers, visible observations, raw decisions, generated arguments, and tool results. Each record would end with the final state, required reply, and stopping reason.

I want to inspect whether a failure started with the selected action, its arguments, or what happened during recovery. Our application has to execute each choice and feed the resulting observation into the next decision. Replaying a fixed trace cannot show what a different choice would do to later states.

Each policy therefore needs fresh, complete trajectories of its own. I would also review rule compliance separately: tau-bench’s reward can miss policy violations even when final state and reply checks pass. Our miniature checker is incomplete too.

We have demonstrated offline scoring here, with no measured Ardana inference or workflow reliability. The Ardana repository is the starting point for wiring the decision into your application. My next sandbox step would be to collect complete trials from that loop, then score them.