Open-Book Exams Written by ChatGPT: Rethinking Student Assessment in the AI Era

Forum topic · 小凯 · 2026-05-18

Summary

A university engineering instructor ran an open-book, take-home exam where students were free to use ChatGPT on one condition: they had to submit their full conversation logs alongside their answers. Analysis of the logs revealed three usage patterns: answer retrieval (pasting exam questions verbatim), guided collaboration (multi-turn dialogue to refine thinking), and critical verification (forming one's own judgment, then checking and correcting the AI's output). The key insight is that in AI-available environments, the cognitive task of assessment shifts from producing solutions to evaluating their validity. A student who can find and fix bugs in ChatGPT-generated code demonstrates genuine understanding. The paper recommends exams test whether students can judge an answer's correctness, identify errors, and repair them—mirroring real engineering work. Caveats include unspecified sample size, possible Hawthorne effects, and the burden that transcript submission itself creates. Source post references Mahmoud (2026, arXiv:2605.12363), Mollick & Mollick (2023, arXiv:2306.10052), and Swauger (2023, Hybrid Pedagogy).

Every engineering instructor knows students use ChatGPT for homework. Rather than pretend this isn't happening, Mahmoud ran an extreme experiment: students were allowed to use ChatGPT freely on an open-book, take-home exam, with a single condition—they had to submit the complete conversation log along with their answers.

This is not a study about "cheating rates," but an observation of "how students think when AI already exists."

Three Usage Patterns

Analyzing the conversation logs revealed three modes:

1. Answer retrieval — the lowest tier. Students paste the exam question into ChatGPT verbatim and submit the answer directly. This produces the least learning value.

2. Guided collaboration — the middle tier. Instead of asking for answers directly, students use multi-turn dialogue to progressively narrow the problem, letting the AI help them organize their thinking and compare options.

3. Critical verification — the highest tier. Students first form their own judgment, then have ChatGPT generate an answer, and then check, correct, and supplement it item by item. When the AI gives wrong or incomplete answers—which happens frequently—students must find and fix the errors themselves. This process reveals the depth of their understanding.

Key Finding

The most important finding: in AI-available environments, the cognitive task of assessment shifts from "producing a solution" to "evaluating the validity of a solution." A student who knows that ChatGPT-generated code has a bug and can locate and fix it has precisely demonstrated understanding of the code—not been replaced by AI.

The paper's recommendation is direct: exams should no longer test "write the correct answer"—they should test "judge whether this answer is correct; if not, where is the error, and how do you fix it?" That is exactly what engineers do in real work—not write code from scratch, but analyze, debug, and improve existing code.

Open Questions

Sample size and disciplinary scope are not specified.
Knowing their transcripts would be reviewed, students' usage behavior may be subject to a Hawthorne effect.
Requiring transcript submission is itself an assessment burden that can be gamed—how do students meet the requirement without cheating?

References

1. Mahmoud, Q. H. (2026). *Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT*. arXiv:2605.12363 [cs.CY]. 2. Mollick, E. R., & Mollick, L. (2023). *Assigning AI: Seven Approaches for Students, with Prompts*. arXiv:2306.10052. 3. Swauger, S. (2023). *Our Bodies Encoded: Algorithmic Test Proctoring in Higher Education*. Hybrid Pedagogy.