Open-Book Exam Where ChatGPT Is Allowed: How Should We Test Students in the AI Era?

Forum topic · 小凯 · 2026-05-18

Summary

A professor ran an experiment allowing students unrestricted use of ChatGPT during an open-book take-home exam, with one condition: students had to submit their complete conversation logs alongside their answers. Analysis of the logs revealed three usage patterns: answer retrieval (pasting exam questions directly and submitting AI output), guided collaboration (multi-turn dialogues to narrow problems and organize thinking), and critical verification (forming independent judgments, generating answers with ChatGPT, then checking, correcting, and refining them). The key insight is that in AI-permitted environments, the cognitive task of assessment shifts from producing solutions to evaluating the validity of solutions. A student who can spot a bug in AI-generated code and fix it demonstrates genuine understanding. The paper recommends exams should test whether students can judge an answer's correctness, identify errors, and repair them—mirroring real engineering work of analyzing, debugging, and improving existing code. Caveats include unreported sample size and discipline scope, potential Hawthorne effects, and the question of whether submitting conversation logs is itself a gamable requirement. References include Mahmoud (arXiv:2605.12363) and Mollick & Mollick (arXiv:2306.10052).

Every engineering instructor knows students use ChatGPT for homework. Rather than pretend this isn't happening, Mahmoud ran an extreme experiment: students were allowed to use ChatGPT freely during an open-book take-home exam, with one condition—they had to submit the complete conversation logs along with their answers.

This is not a study about cheating rates, but an observation of how students think when AI already exists.

Key Points

Analyzing the conversation logs revealed three usage patterns:

Answer retrieval — the student pastes the exam question verbatim into ChatGPT and submits the answer directly. This is the lowest-level use, producing the least learning value.
Guided collaboration — the student doesn't ask for answers directly, but uses multi-turn dialogue to progressively narrow the problem, letting the AI help organize their thinking and compare options.
Critical verification — the student forms their own judgment first, uses ChatGPT to generate answers, then checks, corrects, and supplements each item. When the AI gives wrong or incomplete responses—which happens frequently—students must discover and fix the errors themselves. This process exposes the depth of their understanding.

The most important finding: in an AI-available environment, the cognitive task of assessment shifts from *producing solutions* to *evaluating the validity of solutions*. A student who knows ChatGPT's code has a bug and can locate and fix it has precisely demonstrated they understand the code—not that they've been replaced by AI.

Implications

The paper's recommendation is direct: exams should no longer test "write the correct answer"—they should test "judge whether this answer is correct, and if not, where does it go wrong and how do you fix it." This is exactly what engineers do in real work: not writing code from scratch, but analyzing, debugging, and improving existing code.

Open Questions

Sample size and disciplinary scope are not specified.
Students knew their conversation logs would be reviewed, so their usage behavior may be subject to a Hawthorne effect.
Submitting conversation logs is itself an assessment requirement that can be gamed—how do students complete this requirement without cheating?

References

1. Mahmoud, Q. H. (2026). *Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT*. arXiv:2605.12363 [cs.CY]. 2. Mollick, E. R., & Mollick, L. (2023). *Assigning AI: Seven Approaches for Students, with Prompts*. arXiv:2306.10052. 3. Swauger, S. (2023). *Our Bodies Encoded: Algorithmic Test Proctoring in Higher Education*. Hybrid Pedagogy.