Automation Bias in the AI Act: The Law Knows the Weakness of Its Own Reviewer

Article 14 of the EU AI Act contains a half-sentence you do not expect to find in a statute, because it reads as if a psychologist had helped draft it. The people who oversee a high-risk AI system must be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system”. The law even gives this tendency a name, which it adds in brackets: “automation bias”. An analysis in the European Journal of Risk Regulation notes that it is the only cognitive bias the entire AI Act names.

For HR departments this half-sentence is no legal footnote, because AI that filters applications, evaluates performance or allocates tasks counts as high-risk. For these systems the human oversight obligation applies from 2 December 2027, after the postponement by the Digital Omnibus. Anyone introducing applicant management with an AI ranking today is therefore building the workplace of a person whom the law requires to contradict the machine, and whom the same paragraph certifies as inclined not to do so.

A developer from Cursor, which now belongs to SpaceX, recently explained to students in a lecture how this tendency can be handled differently. In the AI agents he works on, no human approves every single command any more, because a person who nods through hundreds of commands in a row at some point stops checking and simply clicks through, so a second model now does the checking. The engineer and the legislator have made the same observation and drawn opposite conclusions, since the one abolished the tired reviewer, and the other made him mandatory and wrote his tiredness into the law as well.

A safeguard that ships with its own weak spot

That people make poor guardians of well-functioning machines was not discovered by the AI debate. The psychologist Lisanne Bainbridge of University College London described it in 1983 in a five-page paper titled “Ironies of Automation”. In her analysis, automating power plants and factories does not remove the human but leaves him precisely the task he performs worst, namely monitoring a process in which almost nothing ever happens. She drew on vigilance research, according to which even a highly motivated person cannot keep effective attention on a source where hardly anything happens for more than about half an hour.

In 2022, Ben Green of the University of Michigan analysed 41 policies that prescribe human oversight of government algorithms and arrived at an uncomfortable finding. The evidence suggests that people cannot perform this oversight the way the rules imagine, so the obligation mainly creates a false sense of security that shields the algorithms themselves from scrutiny.

I call this situation the Oversight Paradox: a law places a human as the safeguard over a machine and names, in the same article, why this safeguard fails. It is not a slip by the legislator, who at least states the weakness openly, but the result of an assumption that runs through almost every debate about human oversight: that a tendency can be switched off as soon as you know about it.

Why knowledge and experience do not protect

The obvious answer to Article 14 appears in many of the guides that law firms and consultancies are currently publishing, and it is training: reviewers are to learn what automation bias is, how it feels and in which situations it strikes. Companies already gave the same answer to Article 4, which has required AI literacy since February 2025, and it has the same problem there that I described in AI Literacy Is a Construction Problem: it changes knowledge, but not the situation in which people act.

How little knowledge achieves against this tendency is shown by a study that Thomas Dratsch and colleagues at University Hospital Cologne published in the journal Radiology in 2023. A total of 27 radiologists assessed mammograms with the support of a purported AI that deliberately suggested a wrong category in twelve of 40 cases. The most experienced among them, specialists with more than a decade of practice, were right in 82 percent of cases when the suggestion was correct and in only 45.5 percent when it was wrong, and among the inexperienced the rate even fell below 20 percent.

Even more sobering are the experiments that Kathleen Mosier, Linda Skitka and their colleagues ran with pilots and students in NASA studies in the 1990s, because they tested exactly the two recipes that almost every oversight rule relies on today. Two-person crews also succumbed to the tendency, and explicit training to cross-check the automated displays did not remove it either. The four-eyes principle and mandatory training, the two most familiar tools of every compliance department, already proved too weak against this tendency almost thirty years ago.

The third hope, that the AI might simply explain its suggestions, also carries less far than expected. Researchers around Zana Buçinca at Harvard summarised the state of research in 2021 by saying that explanations do not reduce overreliance on AI and, according to some studies, even increase it. An explanation then works like a competence signal that people no longer read but merely register.

The solution is in the same sentence

Read Article 14(4) word by word rather than as a training mandate, and the solution turns up in the grammar. The system is to be provided to the deployer in such a way that the people to whom oversight is assigned “are enabled, as appropriate and proportionate” to fulfil their tasks. Being enabled, however, is not an attitude acquired in a seminar but a property of the environment in which a person works. The accompanying Recital 73 explicitly requires authority alongside competence and training, and an authority to object is unfortunately worth little if objecting costs more time in daily work than agreeing.

At Engaginglab we call this Behavioral Architecture: the designable layer of defaults, friction and context that shapes behavior before a conscious decision sets in. Human oversight is a textbook case, because the law prescribes its result and leaves its construction open. Whoever introduces AI in HR therefore writes, whether they call it that or not, the ‘shift schedule’ of their reviewers, and this shift schedule contains three things: how many cases they see, in what order they judge, and how you can tell whether they are still looking.

First lever: attention as a budget

European airports have drawn a consequence from vigilance research that applies in no office, because whoever screens luggage on the X-ray image may do so for only 20 minutes at a stretch under EU rules and then moves to another position. Nobody considers the screeners poorly trained for that reason; the task is simply considered so tiring that it needs a limit, and this limit sits not in the training binder but in the shift schedule. A dissertation by Daniela Buser at the University of Basel has since refined the rule: the screeners' hit rate declined over time only when they had to inspect many images per minute.

For the oversight of a recruiting AI, it follows that not every case needs the same attention, and the law explicitly requires oversight “as appropriate and proportionate”, that is, in proportion to risk. Oversight becomes productive when the system itself sorts which cases reach the reviewer at all: every proposed rejection, every borderline case in which the AI is uncertain, and every case in which two criteria contradict each other, while clear-cut forwardings are checked by sampling. The developer from Cursor described the same principle for his AI agents, by the way, since there the agent is to disturb the human only once all cheaper routes are exhausted. The reviewer then sees fewer cases, but each with fresh eyes, and the speed for which the AI was introduced is preserved exactly where an error costs little.

Second lever: judge first, then compare

The second lever concerns the order in which a reviewer receives information, and it works because a recommendation that appears first on screen anchors one's own judgment before that judgment could even form. The same researchers around Buçinca tested three variants meant to force people to think for themselves, including one in which participants first decide on their own and only then see the AI's recommendation. These interventions reduced overreliance more clearly than mere explanations but did not eliminate it, and participants gave the most effective variants the worst ratings.

This finding explains why the second lever needs the first. If a reviewer has to assess every case on his own first, he rightly experiences the procedure as a brake and will circumvent it, whereas the same effort is well invested in a rejection or a borderline case. For an application, this means in practice that the reviewer first reads the CV against three criteria for the position defined in advance, only then sees the AI's score, and justifies any deviation between the two in one sentence, whichever way he decides.

Third lever: measuring alertness without spoiling it

The obvious metric for alert oversight is the override rate, the share of cases in which the reviewer decides differently from the AI, and a rate close to zero does indeed suggest a reviewer who just nods along rather than a flawless machine. As a target it is nevertheless useless, because a reviewer measured on his override rate will at some point object in order to meet the rate, and has thereby merely changed the direction in which he is not looking. Using the example of the defensive dashboard, I have described how reliably a metric reshapes the behavior it is only supposed to measure.

Airports have a solution for this too, and it transfers soberly. With Threat Image Projection, the X-ray system inserts the image of a prohibited item that is not actually in the bag into one to four percent of real pieces of luggage, and the screener learns immediately whether he found it. The technology thus measures whether someone is looking and keeps him alert at the same time, because the rare event he is waiting for is suddenly not quite so rare.

Transferred to a recruiting AI, this means mixing fictitious test cases into the stream whose correct answer is known and in which the AI is deliberately wrong, in such a way that all reviewers know such cases exist but none knows which one it is. The hit rate on these test cases is an honest metric because it can only be improved by looking, and it belongs in a regular conversation about working methods rather than in a leaderboard. The same research program around Mosier and Skitka produced the finding that pilots who felt accountable for their strategies of interacting with the automation double-checked the displays more often and made fewer errors. Such a sense of accountability is more likely to arise in a conversation about the path to a decision than in a table of its results.

A recruiting team, rebuilt once

A deliberately simple example shows how the three levers work together. A recruiting team receives 1,200 applications a month, and the new software proposes a category for each, forward or reject. In the first draft, a recruiter clicks through all 1,200 proposals, which the process description lists as human oversight and which, according to everything vigilance research knows, becomes a ritual after half an hour.

In the rebuild, only the proposed rejections, the borderline cases and a sample of the forwardings reach the recruiter. She reads each of these cases first against the three criteria of the position before she sees the score, and two percent of the cases in her pile are fictitious test cases whose results the team discusses together once a month. The software remains as fast as it was sold, the recruiter reviews fewer cases but each of them for real, and the company can for the first time prove with a number that its oversight is looking, instead of merely writing it into the process description.

What the engineer abolished

The agent developers have resolved the Oversight Paradox in their own way by replacing the human with a second machine, and for the commands of a programming tool that may be enough. For decisions about people, about applications, promotions and dismissals, Europe has deliberately chosen the human, and this choice can only be honored if someone builds the workplace in which that human can stay alert.

The legislator has already done the hardest part by writing the reviewer's weakness into the law instead of keeping quiet about it. Only one question remains open, and every HR department can answer it for itself before the next rollout: How many cases does your reviewer see per hour, and when did he last contradict the machine?