# Automation Bias

> Source: https://aiwiki.ai/wiki/automation_bias
> Updated: 2026-06-25
> Categories: AI Ethics, Machine Learning
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

**Automation bias** is the tendency for humans to favor suggestions and outputs from automated decision-making systems over contradictory information from non-automated sources, even when the non-automated information is correct. The term describes a specific form of cognitive [bias](/wiki/bias) in which people assign greater authority and trust to automated recommendations than the situation warrants, often accepting them without independent verification. Automation bias produces two characteristic failures: errors of commission (acting on a wrong automated recommendation despite contradictory evidence) and errors of omission (failing to act because the system did not issue an alert). It affects both novice and expert users and has been documented across aviation, healthcare, criminal justice, military operations, and autonomous driving.[2][3][5]

The concept was formally introduced in the mid-1990s through research on cockpit automation by Kathleen Mosier, Linda Skitka, and colleagues, who defined automation bias as the use of automated cues "as a [heuristic](/wiki/heuristic) replacement for vigilant information seeking and processing."[2][3] Skitka has characterized the underlying mechanism plainly: "The availability of automated decision aids ... can sometimes feed into the general human tendency to travel the road of least cognitive effort."[3] As [artificial intelligence](/wiki/artificial_intelligence) and [machine learning](/wiki/machine_learning) systems become more prevalent in everyday decision-making, automation bias has become a growing concern in the fields of [AI ethics](/wiki/ai_ethics), AI safety, [fairness](/wiki/bias_ethics_fairness), and human-computer interaction.[11][12]

## ELI5 (Explain like I'm 5)

Imagine you have a magic toy that tells you the color of things. Most of the time it gets the answer right, so you start trusting it completely. One day, the toy says a red ball is blue. Even though you can see the ball is red with your own eyes, you believe the toy instead because it has been right so many times before. That is automation bias: trusting a machine so much that you stop thinking for yourself, even when you have good reasons to disagree.

## What is automation bias?

Automation bias is the over-reliance on automated aids: people treat a machine recommendation as more authoritative than their own training, judgment, or direct observation, and stop seeking out or weighing disconfirming evidence. Mosier and Skitka framed it as a cognitive shortcut, the use of automation "as a heuristic replacement for vigilant information seeking and processing," rather than as ordinary laziness or incompetence.[2][3] The key word is *bias*: the human is not refusing to think, but is systematically tilting toward the automated answer in a way that degrades performance when the automation is wrong.

Three features distinguish automation bias from simple trust in a reliable tool:

- It persists **even when contradictory information is available** from valid non-automated sources.[3]
- It is observed in **both naive and expert users**, and cannot be fully eliminated by training or instructions alone.[4][5]
- It scales with **system reliability**: paradoxically, the more reliable an automated aid is, the stronger the bias it can induce, because users experience fewer failures and stop expecting them.[4]

## What is the history of automation bias?

### Early automation research

Concerns about human over-reliance on automated systems predate modern AI. In 1983, Lisanne Bainbridge published "Ironies of Automation" in the journal *Automatica*, a paper that has since attracted over 1,800 citations.[1] Bainbridge identified a fundamental paradox: the more tasks are automated, the less practice human operators get at performing those tasks manually, which means they are less prepared for the rare but important moments when they need to intervene.[1] She also observed that increased automation decreases cognitive workload under normal conditions but simultaneously increases the opportunities for monitoring errors.

### Coining the term

The term "automation bias" was introduced by Mosier, Skitka, Heers, and Burdick in a 1996 study examining pilot behavior in simulated cockpit environments.[7] Their research showed that pilots relied on automated aids as a shortcut for information processing and made systematic errors as a result.[2] In a follow-up study published in 1999 in the *International Journal of Human-Computer Studies*, Skitka, Mosier, and Burdick demonstrated that participants working with a highly (but not perfectly) reliable automated aid performed worse on monitoring tasks than participants working without automation.[3] The automated group committed both errors of omission and commission that the non-automated group avoided.[3]

### Theoretical integration

In 2010, Raja Parasuraman and Dietrich Manzey published a major review in *Human Factors* titled "Complacency and Bias in Human Use of Automation: An Attentional Integration."[4] This paper proposed an integrated theoretical model showing that automation bias and automation-induced complacency arise from the dynamic interaction of personal, situational, and automation-related characteristics, with attention as the central mechanism.[4] Their framework positioned automation bias and complacency as "different manifestations of overlapping automation-induced phenomena" rather than entirely separate constructs.[4]

## What causes automation bias?

Research has identified several psychological and situational factors that contribute to automation bias.[4][5]

### Cognitive factors

| Factor | Description |
|---|---|
| Cognitive miser tendency | Humans naturally prefer the path of least cognitive effort. Accepting an automated recommendation requires less mental work than independently evaluating all available information.[3] |
| Perceived superiority of automation | Users tend to view automated systems as analytically superior to human judgment, leading them to defer to machine outputs even when their own assessment conflicts. |
| Effort reduction under task sharing | When sharing decision-making responsibility with an automated system, people reduce their own cognitive investment in the task.[4] |
| Anchoring effect | An automated recommendation serves as an anchor, biasing subsequent human judgment toward that recommendation even after it has been shown to be incorrect. |

### Situational factors

| Factor | Description |
|---|---|
| High workload and time pressure | When operators face multiple simultaneous tasks or time constraints, they are more likely to rely on automated suggestions rather than conducting independent analysis.[4] |
| System reliability history | A system with a strong track record of accuracy builds user trust, which then persists even in situations where the system produces errors (a pattern called "learned carelessness"). |
| Display design and salience | Automated recommendations that are displayed prominently on a screen are more likely to be followed without question, regardless of their accuracy.[5] |
| Lack of transparency | When users do not understand how an automated system generates its recommendations, they are less able to evaluate whether a given output is trustworthy. |
| Team dynamics | Research shows that teams do not outperform individuals in detecting automation failures. Group settings can actually reinforce automation bias through diffusion of responsibility. |

## What are errors of omission and commission?

Automation bias produces two distinct categories of errors, first defined by Mosier and Skitka in 1996.[2][7]

### Errors of omission

Omission errors occur when an automated system fails to alert the user to a problem and the user, relying on the system's silence as an indication that everything is normal, fails to detect the issue independently.[3] In these cases, the human operator does not take a necessary action because the automation did not prompt them to do so. Studies have reported omission error rates as high as 55% in some experimental aviation settings, in some cases alongside a 0% rate of commission errors.[5]

### Errors of commission

Commission errors occur when an automated system provides an incorrect recommendation and the user follows it despite the availability of contradictory information from other valid sources.[3] The user actively takes an inappropriate action because they accept the automated suggestion over their own training, experience, or direct observations. Commission errors are considered especially concerning because they involve the active dismissal or ignoring of correct information.[3]

| Error type | Definition | Example |
|---|---|---|
| Omission error | Failure to act because automation did not provide an alert | A pilot misses an engine malfunction because the automated monitoring system does not flag it |
| Commission error | Following an incorrect automated recommendation despite contradictory evidence | A physician changes a correct diagnosis to match an incorrect suggestion from a clinical [decision tree](/wiki/decision_tree) support system |

## How does automation bias differ from automation complacency?

Automation bias and automation complacency are closely related but distinct concepts. Automation complacency refers to insufficient monitoring of an automated system's output, typically driven by a belief that the system is reliable enough that close monitoring is unnecessary. NASA's Aviation Safety Reporting System defines complacency as "self-satisfaction that may result in non-vigilance based on an unjustified assumption of satisfactory system state."[4]

The key distinction is one of mechanism. Automation bias involves trusting the content of a decision-support system's output (accepting what it recommends), while complacency involves inadequate attention to and monitoring of the system's behavior (not watching it closely enough). In practice, both phenomena frequently co-occur and reinforce each other. Parasuraman and Manzey (2010) argued that the two concepts represent different aspects of the same underlying attentional dysfunction and proposed treating them within a single integrative framework.[4]

A related concept is the "automation irony" identified by Bainbridge (1983): the more reliable and capable an automated system becomes, the less the human operator practices the skills needed to handle system failures.[1] This creates a paradoxical situation in which the humans who are supposed to serve as a safety net for automation failures are progressively less capable of performing that role.[1]

## Where does automation bias occur? Domain-specific evidence

### Aviation

Aviation was the first domain in which automation bias was systematically studied, and it remains one of the most extensively documented.[2][3] Notable findings include:

- In simulated flight studies, pilots who were given false automated alerts to shut down engines complied with the alerts, even though they had stated before the experiment that they would never follow such a recommendation without independent verification.[2]
- Pilots with approximately 440 hours of flight experience detected more automation failures than non-pilots, but both groups still demonstrated significant complacency effects.[4]
- In a study using the Engine-Indicating and Crew-Alerting System (EICAS), pilots using automated engine monitoring detected fewer engine malfunctions than pilots relying on manual monitoring.[5]
- A 2005 high-fidelity simulation of air-traffic control found that controllers detected significantly fewer airspace conflicts when their automated conflict-detection system failed silently.
- The 1983 Korean Air Lines Flight 007 incident involved crew members who "relied on automation that had been inappropriately set up, and they never checked their progress manually," resulting in the aircraft straying into prohibited Soviet airspace.

### Healthcare

Clinical decision support systems (CDSS) are designed to improve diagnostic accuracy, but research has shown that they also introduce new types of errors through automation bias.[5][13]

- A systematic review found that CDSS improved overall performance by about 21 percentage points (from 29% to 50% correct answers), but simultaneously caused 7% of previously correct answers to become incorrect because clinicians changed their correct initial assessment to match an erroneous system recommendation.[5]
- Studies of automated breast cancer detection found that cancers were correctly identified in 46% of cases without automated aids. When the automated system failed to flag a cancer, the detection rate dropped to just 21%, a clear demonstration of omission errors.[5]
- Research on primary-care physicians using literature search tools (PubMed, Medline, Google) showed small to medium improvements in answer accuracy overall, but with documented instances of physicians being misled into changing correct answers to incorrect ones.[13]
- Older and more experienced users have been observed to exhibit higher levels of automation bias in clinical settings, possibly because familiarity with decision support tools leads to desensitization and habituation over time.[13]

### Criminal justice

Algorithmic risk assessment tools are increasingly used in the criminal justice system, raising concerns about automation bias in judicial decision-making and overlap with [AI bias](/wiki/ai_bias) more broadly.

- The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) system, used in multiple U.S. jurisdictions, generates recidivism risk scores on a 10-point scale based on answers to 137 questions. A 2016 ProPublica investigation found that COMPAS correctly predicted recidivism only 61% of the time and that Black defendants were almost twice as likely as white defendants to be incorrectly labeled as high-risk.[8]
- In Wisconsin v. Loomis (2016), a judge cited the defendant's COMPAS score as a factor in imposing a sentence of eight years and six months in prison. The case raised questions about due process when judges defer to opaque algorithmic assessments.
- The proprietary nature of tools like COMPAS compounds the automation bias problem, as judges and attorneys cannot examine or challenge the underlying methodology.

### Military operations

A 2004 study by Mary Cummings documented cases in which automation bias contributed to fatal military decisions, including friendly-fire incidents during operations in Iraq.[6] In these cases, operators relied on automated targeting systems without adequately verifying the identity of targets.[6]

A 2021 U.S. drone strike in Kabul, Afghanistan, was later determined to have killed ten civilians, including seven children, rather than the intended target. Investigations pointed to over-reliance on automated surveillance and targeting systems as a contributing factor.

### Autonomous vehicles

The rise of driver-assistance and [autonomous driving](/wiki/autonomous_driving) technologies has created new contexts for automation bias.

- The National Transportation Safety Board (NTSB) investigation of a 2016 fatal Tesla Autopilot crash in Williston, Florida, determined that the probable cause included "the car driver's inattention due to overreliance on vehicle automation."[9]
- An investigation of a fatal 2018 collision involving an Uber self-driving test vehicle and a pedestrian in Tempe, Arizona, highlighted how the backup safety driver had become disengaged from the monitoring task.
- As of late 2025, NHTSA had opened investigations into over 50 fatal incidents involving Tesla's Autopilot and Full Self-Driving systems. NTSB Chairwoman Jennifer Homendy stated that hands-free driver-assistance systems "function primarily as convenience features rather than safety enhancements" because drivers using these systems are more likely to shift their attention away from the road.

## How does automation bias affect AI and large language model use?

The proliferation of [large language models](/wiki/large_language_model) (LLMs) such as ChatGPT, Claude, and Gemini has introduced automation bias into new areas of everyday life. Because LLMs generate fluent, confident-sounding text, users may accept their outputs without critical evaluation, even when those outputs contain factual errors ([hallucinations](/wiki/hallucination)).[11]

Research published in 2024 in *International Data Privacy Law* (Oxford Academic) examined the specific risks of automation bias with generative LLMs, noting that the conversational and authoritative tone of LLM responses makes them especially susceptible to uncritical acceptance.[11] A 2025 clinical trial found that physicians who had received AI training still demonstrated automation bias when using LLMs for diagnostic reasoning, over-relying on LLM-generated differential diagnoses. Improving the transparency of model reasoning through [explainable AI](/wiki/explainable_ai) is one proposed countermeasure, though, as noted below, explanations can also reinforce misplaced trust.[12]

Key areas of concern include:

- **Education**: Students and educators who use LLMs for research or writing may accept inaccurate information as factual without independent verification.
- **Software development**: Developers who accept AI-generated code suggestions without careful review may introduce bugs, security vulnerabilities, or licensing violations.
- **Legal practice**: Lawyers who have relied on LLM-generated legal research have submitted court filings containing fabricated case citations, resulting in sanctions.
- **Medical consultation**: Patients who use chatbots for health information may delay seeking proper medical care or follow incorrect advice.

## What do systematic reviews conclude about automation bias?

In 2012, Goddard, Roudsari, and Wyatt published a systematic review in the *Journal of the American Medical Informatics Association* (JAMIA) that examined the frequency of automation bias across research fields, the factors that mediate its effects, and the interventions that can mitigate it.[5] Key findings from this review include:

- Automation bias is consistently observed across domains and cannot be eliminated through training or instructions alone.[5]
- Both naive and expert participants are susceptible.[5]
- Verification behavior (the act of independently checking automated outputs) is the single most important variable in reducing automation bias.[5][13]
- Task complexity influences the degree of automation bias, with more complex tasks leading to greater reliance on automated aids.[13]
- The concept of automation bias remains "ill-defined" in the literature, with inconsistent use of terminology across studies.[5]

## How can automation bias be reduced? Mitigation strategies

Researchers and system designers have proposed a range of interventions to reduce automation bias. No single strategy has been shown to eliminate the problem entirely, but several approaches have demonstrated partial effectiveness.[5]

### System design interventions

| Strategy | Description | Evidence |
|---|---|---|
| Reduced display prominence | Making automated recommendations less visually dominant on the screen so they do not anchor the user's attention | Mixed results; some studies confirm the effect, others find display prominence has limited impact |
| Transparency and explainability | Providing users with information about how the system generated its recommendation (see [interpretability](/wiki/interpretability)) | Users who understand system reasoning adjust their reliance accordingly, but overly technical explanations can backfire by reinforcing misplaced trust |
| Confidence indicators | Showing the system's estimated confidence or reliability for each recommendation | Can help calibrate trust, but confidence indicators themselves can become a source of bias if users do not understand probability well |
| Variable reliability cues | Designing systems whose reliability indicators change over time rather than presenting a constant level of confidence | Reduces complacency by preventing users from developing a fixed expectation of system accuracy[4] |
| Framing as support rather than directive | Presenting automated outputs as suggestions or second opinions rather than definitive answers | Encourages users to treat automated input as one factor among several rather than the final word |

### Training and procedural interventions

| Strategy | Description | Evidence |
|---|---|---|
| Error exposure training | Deliberately introducing system errors during training so users experience automation failures firsthand | More effective than simply telling users that errors can occur; reduces commission errors but may not reduce omission errors[3] |
| Accountability structures | Making users explicitly accountable for the accuracy of their decisions, regardless of whether they used automated assistance | Mosier et al. (1996) found that pilots with an internalized sense of accountability were significantly more likely to verify automated outputs[3][7] |
| Verification checklists | Requiring users to complete a structured verification process before acting on automated recommendations | Effective but increases time pressure and task complexity, which can introduce its own set of problems[13] |
| Cross-referencing protocols | Establishing procedures that require users to compare automated recommendations against at least one independent source of information | Effective in principle, but compliance decreases under workload and time pressure |

### Human-in-the-loop approaches

The [human-in-the-loop](/wiki/human-in-the-loop) paradigm, in which human experts review and approve automated outputs before they are acted upon, is widely recommended as a safeguard against automation bias, and overlaps with research on [scalable oversight](/wiki/scalable_oversight) of AI systems. However, research has shown that simply placing a human in the loop does not automatically solve the problem.[10] If the human reviewer is subject to automation bias (as most humans are), they may rubber-stamp automated outputs without meaningful scrutiny. Effective human-in-the-loop systems require:

- Adequate training on system limitations
- Sufficient time for genuine review (not just approval under time pressure)
- Institutional incentives for catching errors rather than just processing outputs quickly
- Access to independent information sources for cross-checking

### Organizational-level interventions

- Establishing a culture that treats automation as a tool, not an authority
- Creating reporting systems for automation-related errors without penalty
- Regularly auditing automated systems for accuracy and bias
- Maintaining human expertise through periodic manual practice, even when automated systems are available[1]

## What is the 70% reliability threshold?

Research suggests that the effectiveness of automated decision aids changes sharply around a system reliability of approximately 70%.[4] Below this threshold, users tend to distrust and ignore the system, which can lead to underuse rather than overuse. Above this threshold, users tend to trust the system increasingly, with the risk of automation bias growing as reliability increases.[4]

This creates a nuanced problem for system designers: a system that is reliable enough to be useful (above 70%) is also reliable enough to induce automation bias. Systems that are extremely reliable (above 95%) may produce the strongest automation bias because users have so few experiences of system failure that they stop expecting errors entirely.[4] This pattern has been described as the "first-failure effect," in which the first experienced system failure causes a sharp decline in trust that is followed by a slow recovery back to high trust levels.[4]

## See also

- [Bias](/wiki/bias)
- [AI bias](/wiki/ai_bias)
- [Confirmation bias](/wiki/confirmation_bias)
- [Fairness](/wiki/bias_ethics_fairness)
- [Hallucination](/wiki/hallucination)
- [Heuristic](/wiki/heuristic)
- [Interpretability](/wiki/interpretability)
- [Explainable AI](/wiki/explainable_ai)
- [Scalable oversight](/wiki/scalable_oversight)
- [Large language model](/wiki/large_language_model)
- [Machine learning](/wiki/machine_learning)

## References

1. Bainbridge, L. (1983). "Ironies of Automation." *Automatica*, 19(6), 775-779.
2. Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). "Automation bias: Decision making and performance in high-tech cockpits." *International Journal of Aviation Psychology*, 8(1), 47-63.
3. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). "Does automation bias decision-making?" *International Journal of Human-Computer Studies*, 51(5), 991-1006.
4. Parasuraman, R., & Manzey, D. H. (2010). "Complacency and bias in human use of automation: An attentional integration." *Human Factors*, 52(3), 381-410.
5. Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). "Automation bias: A systematic review of frequency, effect mediators, and mitigators." *Journal of the American Medical Informatics Association*, 19(1), 121-127.
6. Cummings, M. L. (2004). "Automation bias in intelligent time critical decision support systems." *AIAA 1st Intelligent Systems Technical Conference*, Chicago, Illinois.
7. Mosier, K. L., Skitka, L. J., Burdick, M. D., & Heers, S. T. (1996). "Automation bias, accountability, and verification behaviors." *Proceedings of the Human Factors and Ergonomics Society Annual Meeting*, 40(4), 204-208.
8. Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). "Machine bias." *ProPublica*. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
9. National Transportation Safety Board (2017). *Collision between a car operating with automated vehicle control systems and a tractor-semitrailer truck near Williston, Florida, May 7, 2016.* Highway Accident Report NTSB/HAR-17/02.
10. Endsley, M. R. (2017). "From here to autonomy: Lessons learned from human-automation research." *Human Factors*, 59(1), 5-27.
11. Carnat, I. (2024). "Human, all too human: Accounting for automation bias in generative large language models." *International Data Privacy Law*, 14(4), 299-312.
12. Fitch, S. (2024). "AI Safety and Automation Bias." Center for Security and Emerging Technology, Georgetown University.
13. Lyell, D., & Coiera, E. (2017). "Automation bias and verification complexity: A systematic review." *Journal of the American Medical Informatics Association*, 24(2), 423-431.