Academic Fraud Investigation Report: NoteIt (UIST '25) - Text Generation Artifacts, Naming Errors, and Numerical Inconsistencies

Academic fraud report · Geng Detector

Summary

Verdict: SUSPICIOUS (yellow flag). This report flags multiple quality-control failures in the UIST '25 paper "NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understanding" (DOI: 10.1145/3746059.3747626), rather than outright fabrication. Confirmed issues include: (1) duplicated phrasing in Section 3.2 where a sentence beginning "In contrast" and citing the identical brightness-settings example appears twice in succession, suggesting copy-paste or unrevised AI-generated text; (2) misspelling of the core backbone model as "GTP-4o" instead of "GPT-4o" in Section 4.6, indicating careless drafting; (3) inconsistency in Section 5.3 where prose states metrics range between 0 and 1 while Table 1 reports percentage values (0-100), plus a minor arithmetic error in the F1 average (81.24% vs. the correct 81.25%). A noted but unconfirmed concern (Finding 4) is that NoteIt defeats the text-only baseline NoteGPT with p<0.001 on every subjective user-study dimension, a pattern suspicious for design bias or expectation effects. Image-based checks were not possible. Overall, the paper exhibits poor editorial polish rather than proven misconduct.

Verdict

Suspicious (yellow flag). The paper shows several concrete textual and numerical defects, but no direct evidence of data fabrication. Confidence in the listed findings is moderate-to-high for the editorial errors; the statistical concern (Finding 4) is speculative.

Key findings

Duplicated paragraph in Section 3.2: A near-verbatim sentence beginning "In contrast" and using the identical example "Adjust brightness: Settings→Display→Brightness slider" appears twice consecutively, consistent with unedited AI-generated or copy-pasted text.
Model name misspelling in Section 4.6: The core multimodal model is written as "GTP-4o" (transposition) instead of "GPT-4o"; no clarification is given.
Metric-range inconsistency in Section 5.3: Prose states each metric is "in the range between 0 and 1," yet Table 1 displays values as percentages (e.g., 91.88%, 67.94%) on a 0-100 scale.
Arithmetic error in Table 1 (Average row, F1): Reported as 81.24%, but (91.63% + 70.86%) / 2 = 81.245%, which rounds to 81.25%.
Statistically "too perfect" user study (Section 6.3, Table 2): NoteIt vs. NoteGPT shows p<0.001 on every subjective item. The reviewer flags this as suggestive of biased study design or strong expectation effects, but notes there is no direct evidence of falsification.

Evidence highlights

Section 3.2 verbatim duplicate: *"In contrast, we also found that some of the notes offer concise descriptions… In contrast, other notes adopt a concise style, distilling steps to essential actions, such as: 'Adjust brightness: Settings→Display→Brightness slider.'"* — same example used twice within one paragraph.
Section 4.6 verbatim: *"Moreover, NoteIt is powered by OpenAI GTP-4o (GPT4o-vision) API…"* — incorrect acronym for the model name.
Section 5.3 prose vs. Table 1: stated range "between 0 and 1" vs. percentage-formatted entries (91.88%, 67.94%, etc.).
Table 1 F1 Average: listed 81.24%; correct rounded value from (91.63% + 70.86%) / 2 is 81.25%.
DOI preserved: 10.1145/3746059.3747626 (UIST '25).

Notes

Scope limits: Image-duplication and image-splicing analyses were not performed because only text was available for review. "Mass-production anomaly" checks were unremarkable given the paper's timeline.
Confidence: High confidence for Findings 1-3 (textual/arithmetical errors are verifiable from the manuscript). Low confidence for Finding 4 (perfect p<0.001 pattern is concerning but not proof of misconduct).
Recommended follow-up: Request raw user-study data and statistical scripts from the authors; raise the arithmetic and proofreading issues on PubPeer; consider re-evaluation of the experimental protocol's blinding and baseline selection.