Integrity Review: VeReMi Extension — A Dataset for Comparable Evaluation of Misbehavior Detection in VANETs (Kamel et al., ICC 2020)

Academic fraud report · Geng Detector

Summary

This report evaluates a 2020 IEEE ICC paper that introduces the VeReMi Extension, a simulation-based dataset for benchmarking misbehavior detection in VANETs. The overall verdict is 'questionable but likely benign.' The primary issue is an internal naming inconsistency: Table I defines the 14h–16h afternoon dataset as 'Attack1415,' while Table III reports evaluation results for that same time slot using the suffix '1416' (e.g., DoSRandom1416, ConstPos1416). This inconsistency most likely reflects a typographical or copy-editing error rather than systematic fabrication, though it is especially problematic for a paper whose central contribution is reproducibility. A second concern—suspiciously 'perfect' metric values (e.g., 0.9994, 0.9876)—was investigated and cleared: precision, recall and F1 in Table III were internally consistent (verified by manual F1 computation on DoSRandom1416) and the regularity is plausible given millions of deterministic OMNeT++/SUMO simulation messages. Timeline checks (IEEE 2020 conference, HAL 2021 preprint, tooling versions) raised no anachronisms. Image-based reuse checks were not possible because only text was available.

Verdict

Questionable — likely an editorial/naming error, not systematic fraud. The paper introduces a simulation dataset for VANET misbehavior detection. No evidence of fabrication was found, but an internal label mismatch between the dataset-definition table and the results table is documented and may impede reproducibility.

Key findings

Internal naming inconsistency between Table I and Table III. Table I maps the 14h–16h time slot to the dataset identifier Attack1415, whereas Table III lists evaluation results for the same 14h–16h slot under the suffix 1416 (e.g., DoSRandom1416, RandomPos1416, ConstPos1416).
Severity: 🟡 Likely a copy-editing or versioning mistake, but particularly unfortunate in a paper whose stated goal is to provide a reproducible, comparable benchmark.
'Perfect' metrics cleared. Precision/recall/F1 values in Table III are unusually regular (four decimal places, near-unity). Manual verification of DoSRandom1416 (Precision = 0.9993, Recall = 0.9996 → F1 ≈ 0.99945, rounded to 0.9995) is mathematically self-consistent. The regularity is consistent with deterministic large-scale simulation (OMNeT++/SUMO; Table I indicates message volumes in the millions).
Timeline clean. ICC 2020 conference, HAL preprint dated 2021-02-23, and references to VEINS/OMNeT++/SUMO and ETSI/IEEE standards all predate the publication. No future-tech anomalies.
Image reuse checks not possible. The submitted file is text-only; Figures 1–2 and simulation screenshots were unavailable, so pixel-level duplication and splicing checks (Detections 1 and 3) could not be performed.

Evidence highlights

Table I (p. 5): Explicit mapping Time 14h–16h → Dataset Id: Attack1415.
Table III (p. 6): Results for the 14h–16h slot labeled DoSRandom1416, RandomPos1416, ConstPos1416, etc.
Consistency test: For DoSRandom1416, F1 = 2·(0.9993·0.9996)/(0.9993+0.9996) ≈ 0.99945 → 0.9995 as reported.
DOI: 10.1109/ICC40277.2020.9149132
Conference/year: IEEE ICC 2020; HAL preprint submitted 2021-02-23.

Notes

The label mismatch (1415 vs 1416) is the most actionable finding. Resolution is best pursued via an author clarification or erratum (e.g., on PubPeer), and by cross-checking the official VeReMi-Dataset GitHub repository for the canonical file naming convention. Given the absence of other irregularities, an institutional misconduct complaint is not currently warranted; an erratum/corrigendum is the proportionate remedy.
Limitations: this review is based on text-extracted content only; no figures were available for visual analysis. Findings are limited accordingly, and false negatives/positives remain possible.