A passing test is not a proof
Coding agents can write software that looks right and passes the tests someone thought to run. That still leaves the bugs nobody anticipated.
Formal verification sets a harder bar. The agent must produce code and a machine-checkable proof that the code follows a written specification for every allowed input. A new benchmark called Vero asks whether today's agents can do that across a whole repository, not just one neat function.
The answer is mixed. The strongest tested setup completed 27 of 43 repositories when it had to write both the implementation and the proofs. Ten repositories defeated every configuration in both test modes.
This is real progress. It is also a useful reminder that finishing most of the small obligations does not mean the software as a whole is verified.
The benchmark made the agents keep a whole codebase consistent
Vero contains 43 multi-module projects drawn from real repositories in Python, Dafny, Verus and Coq. The researchers translated them into Lean 4, a language in which software claims can be checked by a proof assistant.
Together, the projects contain 743 scored APIs and 2,705 specifications. They range from ordinary data structures to cryptographic protocols and distributed systems. Definitions and rules are spread across files, so changing one implementation can break proofs somewhere else.
The agents worked in two modes. In proof-only mode, they received reference code and had to prove it. In code-and-proof mode, they had to write the code as well. Every run had tool access, including the file system, build commands and the Lean toolchain, with a 90-minute budget.
The grader rebuilt each answer in a clean project and rejected shortcuts such as hidden axioms. Agents could also prove that a specification was impossible or that reference code was wrong. That audit route caught several defects during the benchmark's own curation.
One configuration was far ahead
The study evaluated Codex with GPT-5.5 at medium and xhigh reasoning, plus Claude Code with Claude Opus 4.8 and Claude Sonnet 5. Each configuration ran in both modes.
GPT-5.5 xhigh completed 27 repositories in code-and-proof mode and 25 in proof-only. Claude Opus 4.8 completed eight and ten. GPT-5.5 medium completed two and six, while Claude Sonnet 5 completed two in each mode.
The gap was not simply about solving isolated proof steps. GPT-5.5 xhigh passed 87.3% of the individual specifications in code-and-proof mode. Yet 16 repositories remained incomplete because a fully verified repository needs every relevant obligation to close and the project still has to build.
No weaker setup finished a repository that the strongest one missed in both modes. Combining the agents therefore added some individual proofs, but no extra completed repository.
The missing work was shared work
Completed repositories were usually organised around reusable helper theorems. In 80 of the 82 full solves across all runs, at least one helper supported two or more specifications. Nearly three quarters of the proof text sat inside these shared lemmas.
That is where the harder runs broke down. Agents often attacked one failing proof at a time. They did not always discover the common invariant that could unlock several files together.
Implementation freedom helped occasionally. In five cases, an agent replaced a difficult reference algorithm with a simpler but still correct version that was easier to prove. In other cases, writing the implementation created new build failures and proof problems.
The cost data points in the same direction, though the authors mark much of it as estimated. About 23% of total evaluation spending went to the ten repositories no setup completed. Unfinished work kept consuming time without producing a verified result.
What is measured, interpreted and still open
Confirmed: the 11-author preprint was submitted on 13 August 2026. The public benchmark and harness are available on GitHub. The paper reports 43 active repositories, four model configurations, two task modes and a 90-minute limit for each run.
Measured in this setup: GPT-5.5 xhigh completed 27 repositories when writing code and proofs together, while ten repositories resisted all eight configuration-and-mode combinations. Individual specification coverage was much higher than full-repository completion.
The researchers' interpretation: repository-scale verification depends on planning a reusable proof structure, not merely solving local goals. Deep chains of shared lemmas were strongly associated with the obligations other agents missed.
Important limits: Vero targets Lean 4, not the full range of programming languages and proof systems. Its repositories are curated translations, the run length may disadvantage slower agents and most recorded cost totals include estimates for runs killed at the time limit.
Still open: how these systems perform after more time, with different proof tools, or inside a real engineering team. A machine-checked repository is a much stronger promise than code that passed a test suite. Current agents can sometimes make that promise. They cannot make it reliably yet.
Sources
- Ye et al. — Vero paper recordPrimary preprint record submitted 13 August 2026. Source for authorship, benchmark scope, headline results and preprint status.
- Ye et al. — full Vero manuscriptFull primary manuscript. Source for experimental setup, model comparisons, proof structure, costs, interpretation and limitations.
- Vero benchmark repositoryAuthors' public benchmark and evaluation harness. Source for the released repository structure, grading safeguards and reproducibility materials.
