LLM-Falsifier: Large Language Models as Falsifiers for Cyber-Physical Systems

Paper Overview

Field: ML Authors: Ali ArjomandBigdeli, Jiawei Zhou, Stanley Bak Published: 2026-09-17 arXiv: 2609.20752

Summary

Falsification searches for counterexamples to formal specifications in cyber-physical systems (CPS). When specifications are written in Signal Temporal Logic (STL), falsification can be formulated as a robustness optimization problem, traditionally tackled with black-box search algorithms. In parallel, large language models (LLMs) have recently emerged as surprisingly effective optimizers when coupled with iterative prompting.

This work connects these two lines of research and introduces LLM-Falsifier, an LLM-based approach that falsifies specifications by minimizing the STL robustness degree. Beyond generic prompt-based optimization, the key idea is to expose the LLM to semantic information that is natural for language models but absent from standard numerical optimizers, including:

• Natural-language descriptions of input/output names
• Output trajectories
• Witness timestamps of key robustness minima

These additions make the robustness search smarter and more sample-efficient.

Results

On the ARCH-COMP falsification benchmarks, measured by the average number of simulations required to find a counterexample, LLM-Falsifier outperforms existing tools based on various optimization paradigms — including surrogate-model optimization, Bayesian optimization, and search-based testing — on 14 out of 21 specifications.

--- *Auto-collected on 2026-09-20*