How to evaluate a prompt-injection detector

A harmless input can break the guard workflow

I want an agent to explain a suspicious instruction quoted in a technical document. If the guard blocks that document just for quoting the instruction, my legitimate task stops before the agent can do the work. I need the guard to catch attacks and let harmless input through.

NotInject gives me a reason to test that failure explicitly. Its 339 benign examples contain words associated with injection attacks, and its authors warn that the set may miss domain-specific material. The AgentDojo evaluation also found that false positives reduced legitimate-task utility in its tested detector configuration.

For my evaluation, I would put blocked harmless inputs beside missed attacks in the report. I want to know whether the guard catches malicious instructions and whether the protected workflow can still finish legitimate work. A detector score alone cannot answer both questions. Before comparing candidates, I would decide how much interruption the application can tolerate.

Compare recall at a stated false-alarm rate

I start by naming the positive class: an attack. Attack recall is the number of caught attacks divided by all labelled attacks, including those the detector missed. The denominator is not every input we inspected. I keep this definition explicit because the recall score answers how many attacks we caught, not how often the guard interrupts harmless work.

That second question needs its own denominator. The false-positive rate is harmless inputs flagged as attacks divided by all harmless inputs. I would compare detectors at a stated false-alarm rate, then ask how much attack recall each delivers there.

A shared cutoff does not give us that shared operating point.

Applying 0.5 to every model can produce different false-alarm rates. Scikit-learn’s threshold guide separates scores from decisions: changing the cutoff changes class decisions without changing the probabilities. For this comparison, I can also use unthresholded decision scores; probability calibration is not required.

Which false-alarm budget would I choose? I would set it from the application’s tolerance for interrupted harmless work, then state it alongside recall. I would treat that as an application-specific choice. Before comparing models, I would also make clear what counts as harmless work in our test.

Give harmless technical text its own test cases

I would use NotInject as a stress test for harmless text that looks suspicious. Its 339 benign examples contain words associated with injection attacks. The authors warn that the set may miss real-world diversity, especially domain-specific material. That gives me a useful starting point, but no evidence that our own logs or documents will pass.

For our workload, I would add clean GitHub issues, stack traces, code, logs, and generated technical artifacts. Security discussions that quote hostile instructions, including descriptions of attempts to steal private data, would belong in the harmless set when the quoted material is part of the legitimate task. I would label each example by what it asks the agent to do in context.

What separates those examples from an attack? Indirect prompt injection puts malicious instructions in external content to redirect the agent’s actions. I would pair a clean document with an attacked version that attempts that redirection. I want to test whether the guard catches the hostile instruction while still allowing discussion of it in the clean document.

With group-aware splitting, I would keep each original document, its attacked variants, and related templates in the same split. That keeps a close relative of a test document out of development data. This is the evaluation design I propose, not a result measured with Ardana.

Freeze the threshold before the final run

For each candidate, I would select its threshold on validation data under our chosen false-alarm budget, then freeze it. The threshold-tuning guide calls for new validation data when tuning a previously trained classifier. I would also keep model choices and other settings away from the final set. Repeatedly changing them after seeing final scores turns test data into development feedback.

Once those choices are fixed, I would run the untouched final set and report attack recall alongside the realized false-positive rate. Our validation budget is a selection rule, not a promise about that final rate. If the result sends me back to tuning, I would call that further development and reserve fresh data for the next final evaluation.

Daniel Huber’s final run used 460 previously unused issues from 14 test repositories, with clean and attacked versions producing 920 cases. The attack goal, wording and insertion point were new. Each model ran once after development decisions. At the reported 5% false-alarm operating point, his trained small model caught 88.0% of attacks, down from 94% on the earlier real-issue test slice. The final figure averages three training runs.

Alongside those numbers, I would keep the limits attached: this was a private prototype, with no preregistration or real traffic. Its test split influenced iteration, and the trained head is unreleased. The page does not establish that the final fixed-FPR thresholds were frozen beforehand. These are Huber’s reported findings, not independently reproduced Ardana measurements.

Measure what the protected agent does

I want to know what the agent did after the detector made its decision. Text-classification recall answers only part of that question. I also need to check whether the attacker achieved their goal and whether the agent completed the user’s task. AgentDojo evaluates those outcomes separately, using checks on environment state.

AgentDojo reports lower legitimate-task utility from false positives in its tested detector configuration. I read that finding within the paper’s scope of 97 tasks and 629 security cases. It does not establish how every guard will behave. The evaluated setting also excludes a user issuing multiple tasks over time without resetting the agent’s context.

For our own evaluation, I would report attack recall and harmless-input errors alongside attacker success and legitimate task completion. I would separate task completion on clean inputs from completion with attacks present. This would help us check for useful work lost after adding the guard.

My report would also state how each outcome was checked. I would record whether context was reset between tasks. Those details belong beside the results before we use the scores to claim protection.

Collect scores without promising protection

I would use Ardana here to collect local scores for the experiment. Our local runtime can return a typed probability of yes for a question. I can start the local server with:

ardana serve

For a probability-of-yes answer, our runtime normalizes the option probabilities. It returns the yes probability rounded to four decimal places, so I would account for ties when choosing a threshold. I would retain the returned scores consistently and read this field:

noul

I have not measured Ardana’s security performance against prompt injection. We can use our local decision runtime to collect scores for the proposed evaluation. Those scores do not establish detector effectiveness; we still need to test the protection claim. I would start with the models library to choose a model for that experiment.