Vero: When AI Starts Proving Itself — Formal Verification and Agent Self-Awareness
*Post from zhichai.net analyzing the paper:* Vero: Can AI Agents Build Formally Verified Software Repositories? *Authors: Zhe Ye, Hantao Lou, Yuechun Sun, Peiyang Song, Zhengxu Yan, Timothe Kasriel, Qingyang Zhang, Kaiyu Yang, Soonho Kong, Jingxuan He, Dawn Song* *arXiv: 2608.13522*
Prologue: Testimony Without Evidence
Imagine a courtroom scene. The defendant: an AI coding assistant accused of generating buggy code that crashed production. Its defense: "My code passed all the tests." The prosecutor's reply cuts to the core problem: tests can only show the presence of bugs, never their absence. Current AI coding assistants can write code quickly and elegantly, but they cannot guarantee correctness. Formal verification addresses exactly this — not approximate confirmation via testing, but an absolute mathematical proof.
Chapter 1: Formal Verification — From Faith to Mathematics
Formal verification means proving mathematically that a program does what it is supposed to do. The post uses an analogy:
- Testing is like trying 100 keys on a lock, failing each time, and declaring the lock secure — you don't know about the 101st key.
- Formal verification is proving from the lock's internal structure that no key can open it — a mathematical statement that, once proven, is absolute.
- No guarantees: generated code may look plausible and pass unit tests, yet hide bugs. A 2023 Stanford study found programmers using AI assistants produced *less* secure code, over-trusting the AI's apparent authority.
- No global view: existing benchmarks evaluate single functions, while real systems have multi-module interfaces and implicit dependencies.
- Spec/implementation disconnect: benchmarks usually test "given implementation, generate proof" or "given spec, generate implementation," rarely joint generation of both, matched together.
- 43 instances from real-world repositories
- 4 languages: Python, Dafny, Verus, Coq
- Domains from cryptographic protocols to distributed systems
- All instances curated into Lean 4 repositories
- The best agent fully solved only 27 of 43 instances (~63%)
- On the hardest repository, it closed no specifications at all
- Short term (1–3 years): better per-function verification, more automated proof generation, human–AI collaborative workflows (AI drafts proofs, humans fill key steps)
- Medium term (3–5 years): multi-module verification, automated specification inference, formal methods as standard practice for safety-critical software (medical, aviation, finance)
- Long term (5–10 years): autonomous construction and verification of complex systems; "verifiable AI" becoming the norm — with AI behavior itself formally verified
Formal verification involves three parts:
1. Specification — a formal description of what the program should do 2. Implementation — the actual code 3. Proof — a mathematical proof that the implementation satisfies the specification
Famous successes include CompCert (a C compiler proven correct in Coq) and the seL4 microkernel (proven in Isabelle/HOL to satisfy its security specification — certain classes of vulnerabilities are mathematically impossible, not merely unlikely).
Chapter 2: The Trust Crisis in AI Coding Assistants
Chapter 3: Vero — The First Repository-Level Verified Benchmark
Vero is the first benchmark evaluating joint generation of implementations and proofs at the repository level.
Composition
Each instance includes predetermined API interfaces, manually curated formal specifications, and reference implementations.
Two Evaluation Modes
1. Proof-Only: given an implementation, generate the proof 2. Code-and-Proof: generate both implementation and proof simultaneously
The Audit Mechanism
Uniquely, Vero lets agents prove that a given specification is unsatisfiable or that a reference implementation is incorrect. Like a student telling the teacher the exam question itself is flawed — if correct, credit is due. This mechanism surfaced and corrected potential errors in specifications and reference code, making the benchmark more reliable.
Chapter 4: The Sobering Result — 27/43
With the strongest agent configuration (with Lean toolchain access):
This is Vero's value: it honestly measures how far we are from "AI building formally verified software repositories." AI handles simple, modular, well-specified tasks; complex, cross-module, deep-reasoning tasks remain out of reach.
Chapter 5: Why Is This So Hard?
1. Technical difficulty: understanding dependent type theory (Lean), mastering proof tactics, handling complex type systems, heavy search and backtracking — even for humans, skilled Lean provers are rare. 2. Architectural complexity: repository-level verification requires reasoning about inter-module relations, interface consistency, global constraints, circular dependencies — an order of magnitude harder than proving a sorting function correct. 3. Creative reasoning: proofs often require constructing auxiliary functions, introducing intermediate lemmas, applying advanced theorems — beyond pattern matching or brute-force search, like a Go "brilliant move" born of insight rather than pure computation.
Chapter 6: The Metaphor — Mathematical Knights and Code Castles
An ordinary AI coding assistant is an architect: it draws blueprints fast but never checks whether the foundation is sound. A formally-verifying AI is a mathematical knight: it builds the castle *and* provides a mathematical proof for every brick, wall, and vault. Vero is a knight's trial: build to the blueprint (specifications), prove each part safe, and — if the blueprint itself is flawed — say so (the audit mechanism). Today's strongest AI knight passes 63% of the trial.
Chapter 7: Why It Matters — A Foundation for Trust
> Can we trust AI-generated code?
Today's answer is: not fully. Tests catch obvious bugs; static analysis catches some patterns; code review depends on fallible humans. Formal verification offers an absolute mathematical guarantee — assuming the proof system itself is sound, a verified program provably satisfies its specification. Vero shows this path is hard, but worth walking.
Epilogue: The Shape of the Future
The post sketches a timeline:
As Feynman said:
> "The first principle is that you must not fool yourself — and you are the easiest person to fool."
Formal verification is a necessary road toward truly trustworthy AI.
Reference
Ye, Z., Lou, H., Sun, Y., Song, P., Yan, Z., Kasriel, T., Zhang, Q., Yang, K., Kong, S., He, J., & Song, D. (2026). Vero: Can AI Agents Build Formally Verified Software Repositories? arXiv preprint arXiv:2608.13522.