A car is cruising on the highway when the accelerator pedal suddenly sticks. The engineer must answer three questions: How severe is this fault (Severity)? How often are drivers exposed to this scenario (Exposure)? Can the driver keep the situation under control (Controllability)?
These three scores combine to determine an ASIL level — from QM (no safety management required) to D (the strictest safety requirements). That level determines how much safety engineering resources the entire downstream development process must invest.
This is Hazard Analysis and Risk Assessment (HARA) under the ISO 26262 standard. It is a core step in automotive functional safety, and getting it wrong can cost lives.
So the question is: can LLMs do this?
The SAFARI Benchmark: 3,000 Real Industrial Cases
SAFARI (Safety-Aware Functional Automotive Risk Inference) is the first industrial benchmark for evaluating LLM-assisted automotive HARA. It contains 3,000 anonymized HARA cases from real industrial practice, covering two task types:
1. Open-ended hazard analysis: given a fault scenario, generate descriptions of possible hazard events 2. Standard-driven risk assessment: assign S/E/C scores and ASIL classifications per ISO 26262
These two tasks are coupled — the output of hazard analysis is the input to risk assessment. If the hazard description is wrong, the downstream risk scoring will necessarily be wrong too. Unlike multiple-choice benchmarks, this is a cascade-sensitive reasoning pipeline.
To evaluate open-ended hazard analysis, the paper proposes the first HARA-specific LLM-as-a-judge protocol, scoring along four dimensions: hazard chain consistency, physical consequence consistency, risk population coverage, and safety goal alignment. This evaluation protocol shows high agreement with certified engineers' judgments.
Nine Frontier Models, All Failing
The paper tested nine frontier LLMs, and the results reveal a clear capability boundary:
What LLMs are good at: generating plausible hazard narratives. Given a "stuck accelerator pedal" scenario, models can produce a professional-sounding hazard description involving loss of vehicle control, potential pedestrian collision, the need for emergency braking, and so on.
What LLMs are bad at: ISO 26262 risk classification. The best model achieves a macro-F1 of only 0.261 on ASIL classification.
For context: ASIL has 5 levels (QM, A, B, C, D), and random guessing yields a macro-F1 of roughly 0.20. The best LLM is only slightly better than chance.
More worryingly, Chain-of-Thought (CoT) prompting not only fails to help, but often makes classification performance worse — in sharp contrast to CoT's benefits on mathematical reasoning and code generation.
Why Does CoT Backfire?
The key insight: HARA risk assessment is not a pure reasoning problem, but a standard-driven judgment problem.
ASIL is deterministically derived from three parameters: S (Severity), E (Exposure), and C (Controllability). These parameters have explicit standard definitions — e.g., Exposure ranges from E0 (incredible) to E4 (high frequency), and Controllability from C0 (controllable in general) to C3 (difficult to control or uncontrollable).
The problem is that judging these parameters requires extensive implicit physical-world knowledge. Is the Controllability of a stuck accelerator on a rain-soaked highway C3 or C2? You need to know: how wet-road friction affects braking distance, whether an ordinary driver in panic can simultaneously manage steering and braking, the probability of surrounding vehicles avoiding the out-of-control car...
CoT makes the model "show its reasoning," but that reasoning rests on incomplete scene understanding. The model may produce a long chain of plausible-sounding analysis while getting the key physical parameter judgments wrong. CoT is not helping the model think clearly — it is helping the model be confidently wrong.
Two Critical Failure Modes
The paper's error analysis pinpoints two main bottlenecks:
1. Omission of scenario-critical context: during hazard analysis, models frequently miss scenario details that affect safety. For example, given a "night-time construction zone," the model may focus only on the core fault ("stuck accelerator pedal") while ignoring how the construction zone limits evasive space. Missing context leads to incomplete hazard descriptions, which then bias the risk assessment.
2. Controllability misjudgment: during risk assessment, models show the highest error rates on Controllability. This makes sense — Severity and Exposure are relatively objective (collision severity and scenario frequency can be inferred from descriptions), but Controllability requires a psychological model of an "average driver": what would a typical driver do when facing a sudden fault? Could they keep control? This is the hardest dimension to quantify.
"Looks Plausible" ≠ "Correct"
The paper reveals a gap worth heeding: LLMs can generate plausible hazard narratives, but cannot reliably perform standard-driven risk classification.
This gap is particularly dangerous. If LLMs were entirely incapable, engineers would not use them. But LLMs "seem capable" — their hazard descriptions read professionally, fluently, and coherently. Someone without deep HARA experience could be misled by this fluency and simply adopt the model's output.
A macro-F1 of 0.261 on ASIL classification means: on safety-level determination, the model is barely better than random guessing. If automotive safety engineers actually used LLMs for ASIL classification and allocated safety resources accordingly — hazards that should be treated as ASIL D might be classified as ASIL B, leading to under-investment in safety with potentially fatal consequences.
A Broader Lesson
SAFARI's findings echo a recent line of research on "LLM unreliability in professional domains":
- Medicine: LLMs write fluent clinical notes but show persistently high error rates in drug-interaction judgment
- Law: LLMs draft legal documents but exhibit systematic bias in sentencing recommendations
- Safety engineering: LLMs write hazard narratives but achieve only 0.261 macro-F1 on ASIL classification
CoT helps with the former (helping the model organize thoughts) but can harm the latter (helping the model be more confidently wrong). This distinction is critical for real-world LLM deployment.
The paper's conclusion is pragmatic: LLMs can serve as HARA assistants — helping engineers draft hazard descriptions and check for missed scenarios. But the final risk judgment must be made by certified engineers. The model is an assistant, not a decision-maker.
---
Paper: SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Code: github.com/xixi47520-hash/HARA
Authors: Chenxi Wu, Zimu Wang, Haiyang Zhang, Wei Wang, Zhijie Xu (Xi'an Jiaotong-Liverpool University)
Standard: ISO 26262 functional safety