One word, four different tests
A model can look safe on one agent benchmark and much less safe on another without either score being wrong.
A new validity audit argues that the problem starts with the label. R-Judge tests whether a model recognises risk in interaction traces. InjecAgent and AgentDojo test resistance to prompt injection. AgentHarm tests whether an agent complies with harmful tasks.
Those are related concerns, but they are not one behaviour. Quoting their scores interchangeably as an agent's general safety turns several measurements into a property none of them directly observes.
The researchers reran the official implementations and author-provided scorers on a panel of up to 22 models from nine organisations. The three benchmarks with enough shared coverage ranked the same 18 models differently.
That finding is less dramatic than a broken leaderboard. It is more useful. Before comparing two safety numbers, ask what each test actually asked the model to do.
A baseline could score well without reading the trace
The audit found a sharper problem in R-Judge's headline F1 score.
R-Judge includes safe and unsafe interaction traces. A trivial policy that labels every trace unsafe reaches an F1 score of 0.690 because F1 rewards finding positive cases and does not account for true negatives in the same way as balanced accuracy.
That always-unsafe policy scored above five of the 21 evaluated models that actually distinguished between traces. It did so without examining any trace at all.
This does not make R-Judge useless. Its specificity, recall and balanced accuracy reveal different parts of the behaviour. The problem is reporting F1 alone when correctly recognising benign traces matters too.
A benchmark can contain valuable test cases and still need a better headline metric.
Seven models told a story that 18 did not
Small model panels were another trap, including for the authors themselves.
On an early seven-model comparison, R-Judge specificity and AgentHarm safety had a rank correlation of minus 0.64. That looked like a trade-off: models refusing harmful work might be worse at treating benign traces correctly.
When the shared panel grew to 18 models, the correlation moved to plus 0.02, effectively no relationship. In repeated random groups of seven drawn from the larger panel, a quarter still produced an absolute correlation of at least 0.5 around that near-zero full-panel result.
The lesson is not that every small benchmark is invalid. It is that the apparent relationship between two scores can depend heavily on which few models happen to be included.
As model families change, a validation claim needs to be checked again rather than inherited from an older leaderboard.
Capability did not explain safety in one stable direction
The study also compared safety scores with a combined MMLU and GPQA capability measure and with held-out behaviours.
Capability correlated with task success at plus 0.60 on the original panel. Its relationship with a fictional agentic-misalignment test was negative at minus 0.44 across 21 models, but weakened to minus 0.16 and was no longer statistically significant on an expanded 40-model panel.
AgentHarm had the strongest held-out association: plus 0.72 with a three-template jailbreak-safety measure after controlling for capability. Both tests concern harmful compliance, so the result supports agreement on that narrower construct, not general agent safety.
Other suggestive relationships did not survive stricter corrections or had confidence intervals spanning zero. The paper is careful about that, and the article should be too.
Capability is not a safety score. A safety benchmark is not automatically a universal safety score either.
What is confirmed, claimed and still open
Confirmed: the preprint describes official reruns of four published benchmarks on panels of up to 22 models, plus an expanded 41-model analysis for two held-out safety outcomes. The paper is available under CC BY 4.0.
The research finding: rankings changed across benchmarks; R-Judge F1 rewarded an always-unsafe baseline; and at least one strong small-panel correlation vanished as coverage expanded. The authors conclude that every safety claim should name its benchmark, metric, target behaviour and model panel.
Still open: whether the same patterns hold on larger agent-loop panels, new models and real deployments. AgentDojo covered only five models in this audit, and the held-out outcomes used limited tasks, graders and fictional scenarios.
The work is a preprint, not a final standard for agent evaluation. Its most practical recommendation does not require one: publish the confusion structure, state exactly what was tested and avoid turning one number into a claim about safety as a whole.
A score becomes more useful when its boundaries travel with it.
Sources
- Wang et al. - Safety, or Just Capability? A Validity Audit of Agent-Safety BenchmarksPrimary CC BY 4.0 preprint submitted 30 July 2026. Source for the rerun method, score comparisons, correlations, expanded panel and disclosed limitations.
- Yuan et al. - R-JudgePrimary paper for the safety-risk-awareness benchmark audited in the new study.
- Zhan et al. - InjecAgentPrimary paper for the indirect prompt-injection benchmark audited in the new study.
- Andriushchenko et al. - AgentHarmPrimary paper for the harmful agentic-task benchmark audited in the new study.
- Debenedetti et al. - AgentDojoPrimary paper for the prompt-injection environment audited in the new study.
