"Add a Confirm Button" Is Not Human Oversight — An AI Oversight Design Blueprint from Dagstuhl Castle

| Item | Details | |------|---------| | Title | Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems | | Authors | Susanne Gaube, Markus Langer, Tim Miller, Kevin Baum, et al. — 20 researchers (Dagstuhl Seminar 25272) | | arXiv | https://arxiv.org/abs/2605.16278 (cs.CY, cs.AI, cs.HC) | | Date | April 2026 | | Core contribution | The first interdisciplinary (CS, HCI, psychology, philosophy, law) framework for human oversight of AI — defining oversight architecture, roles, and processes; providing documentation templates; synthesizing six open research challenges |

The Problem

Imagine you are a hospital director. The board asks you to implement "human oversight" on your radiology AI diagnostic system. You ask: "How, specifically?"

You consult the law — "High-risk AI systems shall be subject to human oversight" (EU AI Act, Article 14). Then you ask the technical experts: "Add a human confirmation button next to the prediction."

If this is human oversight, its only difference from no oversight is one extra click. Button-style oversight is not oversight — it is formal confirmation.

Twenty researchers from computer science, HCI, psychology, philosophy, and law spent a week at Dagstuhl Castle in Germany in the summer of 2025 to address exactly this question: what should the architecture of human oversight of AI actually be?

1. An "Architecture," Not a "Check"

The paper's first key distinction: oversight is not a one-time checking act, but a continuous architecture, defined along three dimensions:

• Temporal: Does oversight occur pre-decision, in-process, or post-hoc? These are architecturally different. "Pre" requires human approval before action (e.g., high-risk surgical devices need physician confirmation first); "post" allows the AI to act first while humans retain the right to intervene (e.g., a license-review system suggesting pass/fail with human reviewers stepping in for edge cases).
• Competency: What can the supervisor do? The paper proposes four core competencies for oversight roles: domain expertise (understanding what AI output means in the field), algorithmic literacy (understanding when AI might fail), skepticism (structural skepticism — knowing when to question, not blindly accepting or rejecting), and self-efficacy (believing one's oversight actions have real effect). Missing any one, oversight becomes token.
• Responsibility: How is legal liability distributed? If a supervisor approves an AI recommendation that leads to patient harm — is the AI developer, the supervisor, or the institution responsible? The paper candidly notes that the legal field has no consistent answer yet.

2. "Roles," Not a Single Person

The paper's deepest contribution: "the supervisor" is not one person — it is a set of roles.

• Trigger — senses anomalies. Not making decisions, but raising alarms. Requires structural skepticism and enough contextual knowledge to know "this result is wrong."
• Verifier — checks the basis of a decision. If the AI says "this is a tumor," the verifier examines where and why — without necessarily making the diagnostic decision themselves.
• Decision-maker — makes the final call. Typically a doctor, judge, or recruiter with domain credentials and legal accountability. Must have the power to veto AI recommendations — and the responsibility for erroneous vetoes.
• Overseer — monitors the system as a whole, not individual decisions: systemic bias against certain groups, rising alarm frequency from triggers (suggesting system degradation). Without this role, all "local oversight" remains isolated.

The paper stresses: these roles may require different people, competencies, and temporal rhythms. Compressing all roles into one person is designing oversight for failure.

3. Writing Oversight Down — Documentation Templates

The paper provides operational documentation templates for institutions deploying AI systems:

• What the system does
• Which roles participate in oversight, each role's competencies and permissions
• The temporal position of oversight: before / during / after decisions
• Thresholds for oversight intervention (what deviation level triggers human involvement)
• Who is responsible for updating the oversight architecture (oversight itself needs oversight — otherwise it ages after system upgrades)

The paper demonstrates the template in healthcare, finance, and HR.

4. Six Open Research Challenges

• Measurability: How do we measure "effective oversight"? If a supervisor vetoes 3% of AI recommendations while raw AI accuracy is 97%, did they defend against 3% errors — or veto some correct conclusions? Is accuracy improvement a good metric of oversight effectiveness? The paper offers no answer.
• Automation paradox: As AI accuracy rises, human supervisors see fewer errors and lose the ability to detect them — precisely in low-error environments where errors carry extreme costs. Aviation calls this "automation surprise." AI research has not yet modeled this paradox.
• Sociotechnical systems: Oversight must place AI systems in their sociotechnical context. A hiring AI overseen in an organization lacking diversity — even with higher veto rates on individual decisions — may still hide group-level bias. Oversight's "field of view" must span individual decision streams.
• Three more: incentive structures (who pays for oversight, who bears its failures), oversight degradation detection (knowing when the architecture ages), and international coordination of oversight standards.

The deepest theme is clear: oversight is a living system, not a button. And almost no one besides this paper treats it as a system to be designed.

My Take

One sentence sums up the framework's value: it completes the transformation of "human oversight" from political slogan to engineering concept.

Until now, "high-risk AI must have human oversight" was a legal statement, an ethical commitment, or a cost barrier — almost never something that could be engineered. A 20-person team spent a week at Dagstuhl Castle doing something foundational and extremely scarce: defining the minimal atoms of an "architecture."

This is basic infrastructure for human-AI coexistence. Without it, "human oversight" is just an excuse for delayed exit.

References

1. Gaube, S. et al. (2026). Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems. https://arxiv.org/abs/2605.16278 2. European Commission. (2021). Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence (AI Act). 3. Parasuraman, R., Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors. 4. Selbst, A.D. et al. (2019). Fairness and Abstraction in Sociotechnical Systems. FAT* 2019.