Imagine a car's accelerator pedal getting stuck during highway cruising. An engineer must answer three questions: How severe is this fault (Severity)? How often are drivers exposed to this scenario (Exposure)? Can the driver maintain control (Controllability)?
The combination of these three scores determines an ASIL rating—from QM (no safety management required) to D (the strictest safety requirements). This rating determines how much safety engineering resources the entire downstream development process needs to invest.
This is Hazard Analysis and Risk Assessment (HARA) under the ISO 26262 standard. It is a core component of automotive functional safety, and errors can cost lives.
So the question arises: can LLMs do this?
The SAFARI Benchmark: 3,000 Real Industrial Cases
SAFARI (Safety-Aware Functional Automotive Risk Inference) is the first industrial benchmark for evaluating LLM-assisted automotive HARA. It contains 3,000 anonymized HARA cases from real industrial practice, covering two task types:
1. Open-ended hazard analysis: given a fault scenario, generate descriptions of possible hazardous events 2. Standard-driven risk assessment: perform S/E/C scoring and ASIL classification of hazards according to ISO 26262
These two tasks are coupled—the output of hazard analysis is the input to risk assessment. If the hazard description is wrong, the downstream risk scoring will necessarily be wrong too. This is completely different from "multiple-choice" benchmarks; it is a cascade-sensitive reasoning pipeline.
To evaluate the quality of open-ended hazard analysis, the paper proposes the first HARA-specific LLM-as-a-judge protocol, scoring along four dimensions: hazard chain consistency, physical consequence consistency, risk population coverage, and safety goal alignment. This evaluation protocol shows high agreement with certified engineers' judgments.
Nine Frontier Models, All Fail
The paper tested nine frontier LLMs, and the results reveal a clear "capability boundary":
What LLMs are good at: generating plausible-looking hazard narratives. Given a scenario like "accelerator pedal stuck," models can write a professionally-sounding hazard description—involving loss of vehicle control, potential pedestrian collisions, need for emergency braking, and so on.
What LLMs are bad at: ISO 26262 risk classification. The best model achieves only a macro-F1 of 0.261 on ASIL classification.
What does 0.261 mean? ASIL has 5 levels (QM, A, B, C, D), and random guessing yields a macro-F1 of roughly 0.20. In other words, the best LLM is only slightly better than random guessing.
Even more concerning: Chain-of-Thought (CoT) prompting not only failed to help, but often made classification performance worse. This stands in sharp contrast to CoT's effectiveness on mathematical reasoning and code generation tasks.
Why Does CoT Backfire?
The key is that HARA risk assessment is not a pure reasoning problem—it is a standard-driven judgment problem.
ASIL is deterministically derived from three parameters: S (Severity), E (Exposure), and C (Controllability). These parameters have explicit standard definitions—for example, Exposure ranges from E0 (incredible) to E4 (high frequency), and Controllability ranges from C0 (controllable in general) to C3 (difficult to control or uncontrollable).
The problem is that determining these parameters requires substantial implicit physical-world knowledge. For example, is the Controllability of "accelerator pedal stuck on a rain-soaked highway" C3 or C2? You need to know: how wet-road adhesion affects braking distance, whether an ordinary driver in a panic state can simultaneously control steering and braking, the probability of surrounding vehicles evading on the highway…
CoT makes the model "show its reasoning," but the model's reasoning is based on incomplete understanding of the scenario. It may produce a long chain of plausible-looking analysis, yet get the key physical parameter judgments wrong. CoT is not helping the model think clearly—it is helping the model be wrong with more confidence.
Two Fatal Failure Modes
The paper's error analysis identifies two main bottlenecks:
1. Omission of safety-critical scene context: during hazard analysis, models frequently omit key details that affect safety. For example, with a "nighttime construction zone" context, the model may focus only on the core fault "accelerator pedal stuck" while ignoring how the construction zone restricts evasion space. The omitted context leads to incomplete hazard descriptions, which in turn biases risk assessment.
2. Controllability misjudgment: during risk assessment, models have the highest error rate on Controllability. This makes sense—Severity and Exposure are relatively objective (crash severity and scenario frequency can be inferred from descriptions), but Controllability requires a psychological model of an "ordinary driver": facing a sudden fault, what would an average driver do? Can they maintain control? This is the hardest dimension to quantify.
"Looks Plausible" ≠ "Is Correct"
This paper reveals a worrying gap: LLMs can generate plausible-looking hazard narratives, but cannot reliably perform standard-driven risk classification.
This gap is especially dangerous. If LLMs were completely useless, engineers wouldn't use them. But LLMs "look competent"—their hazard descriptions read professionally, fluently, and logically. Someone without deep HARA experience could be misled by this fluency and directly adopt the model's output.
A macro-F1 of 0.261 on ASIL classification means: on safety rating determination, the model is barely better than random guessing. If automotive safety engineers actually used LLMs for ASIL classification and allocated safety resources accordingly—some hazards warranting ASIL D treatment might be classified as ASIL B, leading to underinvestment in safety, with potentially fatal consequences.
A Broader Implication
SAFARI's findings echo a recent line of research on "LLM unreliability in professional domains":
- Medicine: LLMs can write fluent medical records, but error rates on drug interaction judgments remain high
- Law: LLMs can draft legal documents, but exhibit systematic biases in sentencing recommendations
- Safety engineering: LLMs can write hazard narratives, but achieve only 0.261 macro-F1 on ASIL classification
CoT is effective on the former (helping models organize their thinking), but may be harmful on the latter (helping models make mistakes with more confidence). This distinction is critical for real-world LLM deployment.
The paper's conclusion is pragmatic: LLMs can serve as auxiliary tools for HARA—helping engineers draft hazard descriptions and check for missed scenarios. But the final judgment on risk assessment must be made by certified engineers. The model is an assistant, not a decision-maker.
---
Paper: SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Code: github.com/xixi47520-hash/HARA
Authors: Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu (Xi'an Jiaotong-Liverpool University)
Standard: ISO 26262 functional safety