# Algorithmic bias

> Source: https://aiwiki.ai/wiki/algorithmic_bias
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: AI Ethics, AI Policy & Regulation, Machine Learning, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Algorithmic bias." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/algorithmic_bias
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Algorithmic bias is the tendency of a computer system to produce systematic, repeatable errors that advantage some groups of people over others. The term covers both a statistical property (a model that is consistently wrong in one direction for one population) and a social consequence (an uneven distribution of jobs, loans, medical care, or police attention). Batya Friedman and Helen Nissenbaum set the reference definition in 1996, writing that they use the term bias "to refer to computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others," where a system discriminates unfairly "if it denies an opportunity or a good or if it assigns an undesirable outcome to an individual or group of individuals on grounds that are unreasonable or inappropriate" [1].

Two features of that definition still do most of the work. Bias has to be systematic: a random glitch that downgrades one applicant is a defect, not a bias. And the grounds have to be inappropriate to the decision: a credit model that denies people with poor repayment histories is doing its job, while one that keys on ethnic surnames is not [1]. Neither requirement mentions intent, which is why algorithmic bias is usually discussed as something a system does rather than something a developer meant.

In [machine learning](https://aiwiki.ai/wiki/machine_learning) the word "bias" is overloaded. It also names the intercept term in a linear model, and the systematic error component in the bias-variance decomposition, neither of which carries any social meaning. This article uses the socially loaded sense, the one that overlaps with [algorithmic fairness](https://aiwiki.ai/wiki/algorithmic_fairness), [AI ethics](https://aiwiki.ai/wiki/ai_ethics), and anti-discrimination law.

## Definition and origins of the term

Friedman and Nissenbaum's paper, published in ACM Transactions on Information Systems in July 1996, predates modern [deep learning](https://aiwiki.ai/wiki/deep_learning) by two decades. They opened with the Sabre and Apollo airline reservation systems and used an automated credit advisor as their running example, then worked through three case studies: the National Resident Match Program for medical residencies, a multilevel scheduling algorithm for timeshare computers, and the British Nationality Act Program [1]. They sorted bias into three categories: preexisting bias, rooted in social institutions, practices, and attitudes; technical bias, arising from technical constraints or design considerations; and emergent bias, which appears only in a context of use, after deployment, when the population or the surrounding practices shift away from what the designers assumed [1].

The [National Institute of Standards and Technology](https://aiwiki.ai/wiki/nist) reworked that taxonomy for AI systems in Special Publication 1270, published in March 2022 by Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall. It names three categories. Systemic bias "result[s] from procedures and practices of particular institutions that operate in ways which result in certain social groups being advantaged or favored and others being disadvantaged or devalued," and need not involve any conscious prejudice. Statistical and computational bias "stem[s] from errors that result when the sample is not representative of the population," arising "from systematic as opposed to random error" and occurring "in the absence of prejudice, partiality, or discriminatory intent." Human bias covers "systematic errors in human thought," including how operators read and act on model output [2].

NIST's framing matters because it rejects the idea that bias is a code defect with a code fix. The report warns against "techno-solutionism," the assumption "that the 'right' code or algorithm can be applied to any problem," and argues that bias in AI has to be treated as a socio-technical problem spanning the dataset, the evaluation regime, and the humans in the loop [2].

## Where bias comes from

| Source | Mechanism | Illustration |
| --- | --- | --- |
| Historical data | The model learns regularities in past human decisions, including discriminatory ones | Amazon's recruiting model penalized resumes containing the word "women's" [7] |
| Representation and sampling | Groups thinly represented in training or benchmark data get worse accuracy | The IJB-A face benchmark was 79.6% lighter-skinned subjects; Adience was 86.2% [8] |
| Proxy labels | The measurable target only stands in for the quantity actually of interest | A health system used spending as a proxy for medical need [14] |
| Measurement | Capture conditions, sensors, or annotation quality differ by group | NIST tied some false-negative differentials to lower-quality border-crossing images [10] |
| Feedback loops | Model output shapes the data the model is later retrained on | Predictive policing can send officers back to the same neighborhoods regardless of the true crime rate [21] |
| Use and interpretation | Operators over-trust or misread the output | NIST groups automation complacency and confirmation bias under human bias [2] |

The proxy-label channel is the least intuitive and often the most damaging, because the model can be accurate at the task it was given and still wrong about the task anyone cared about. [Proxy labels](https://aiwiki.ai/wiki/proxy_labels) are chosen because they are recorded, not because they are correct.

Feedback effects are a separate failure mode that no amount of clean training data prevents. Danielle Ensign, Sorelle Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian formalized the problem for predictive policing in 2017, showing that when patrol allocation is driven by discovered crime, "police are repeatedly sent back to the same neighborhoods regardless of the true crime rate." They found that using citizen-reported incidents rather than officer-discovered ones attenuates the [feedback loop](https://aiwiki.ai/wiki/feedback_loop) but does not remove it without a deliberate correction [21].

## Landmark cases

### COMPAS and the ProPublica analysis

On 23 May 2016 ProPublica published "Machine Bias," an analysis by Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner of [COMPAS](https://aiwiki.ai/wiki/compas), a pretrial risk assessment tool sold by Northpointe Inc. The reporters obtained scores for 18,610 people assessed in Broward County, Florida, in 2013 and 2014, narrowed to 11,757 assessed at the pretrial stage, and followed them for two years [3][4].

| Two-year general recidivism (n = 7,214) | Black defendants | White defendants |
| --- | --- | --- |
| Labeled higher risk, did not reoffend (false positive rate) | 44.9% | 23.5% |
| Labeled lower risk, did reoffend (false negative rate) | 28.0% | 47.7% |

The tool's overall predictive power was modest, with a concordance of 63.6 percent on the binary score and 66.4 percent on the raw 1-10 scale [4]. What drew attention was the asymmetry of the errors rather than the accuracy.

Northpointe answered on 8 July 2016 with a technical report by William Dieterich, Christina Mendoza, and Tim Brennan titled "COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity." It "strongly reject[ed] the conclusion that the COMPAS risk scales are racially biased against blacks," argued that ProPublica had used classification statistics that "did not take into account the different base rates of recidivism for blacks and whites," and maintained that the general and violent risk scales "are equally accurate for blacks and whites" [5]. ProPublica replied on 29 July 2016, conceding a small data-overlap error while holding that black defendants who did not reoffend were about three times as likely as comparable white defendants to be classified high risk once cut points were adjusted [6].

Both descriptions of the data were correct. The disagreement was over which fairness property counted, not over the arithmetic, and the formal results that explain why both could hold at once appeared within months.

### Amazon's abandoned recruiting engine

[Amazon](https://aiwiki.ai/wiki/amazon) began building an automated resume screening system in 2014 that scored candidates from one to five stars. It was trained on ten years of resumes submitted to the company, most of them from men, and it learned to prefer the patterns in that history. The system penalized resumes containing the word "women's" and downgraded graduates of certain all-women's colleges. Amazon discontinued the project in 2017; Reuters reported the episode in October 2018 [7].

The case is cited so often because nothing exotic went wrong. The training data recorded real past decisions, the target label was real, and the model generalized faithfully from both. The system also had no gender field to remove: it recovered the attribute from the wording of the resume itself, which is the standard reason that simply withholding a protected attribute fails on rich text or behavioral data.

### Gender Shades and commercial face analysis

[Joy Buolamwini](https://aiwiki.ai/wiki/joy_buolamwini) and [Timnit Gebru](https://aiwiki.ai/wiki/timnit_gebru) presented "Gender Shades" at the Conference on Fairness, Accountability and Transparency in February 2018. They built the Pilot Parliaments Benchmark (PPB), 1,270 parliamentarians from Rwanda, Senegal, South Africa, Iceland, Finland, and Sweden, labeled by Fitzpatrick skin type and by gender, and used it to audit three commercial gender classification APIs from [Microsoft](https://aiwiki.ai/wiki/microsoft), IBM, and Face++ (built by [Megvii](https://aiwiki.ai/wiki/megvii)) in April and May 2017 [8].

| Gender Shades result | Value |
| --- | --- |
| Error rate on darker-skinned female faces, across all three classifiers | 20.8% to 34.7% |
| Maximum error rate on lighter-skinned male faces | 0.8% |
| Best individual results | Microsoft 0.0% and IBM 0.3% on lighter male faces; Face++ 0.7% on darker male faces |
| Largest gap between the best and worst classified subgroups | 34.4% |
| Male versus female error rate difference | 8.1% to 20.6% |
| Lighter versus darker error rate difference | 11.8% to 19.2% |

The audit also documented why the disparity had gone unnoticed: existing benchmarks were skewed, with 79.6% lighter-skinned subjects in IJB-A and 86.2% in Adience, so aggregate accuracy numbers hid the failure [8].

Inioluwa Deborah Raji and Buolamwini followed up in 2019 with "Actionable Auditing," which tracked what the disclosure actually changed. Within seven months all three audited companies had released new API versions, and error on the darker-skinned female subgroup fell by 17.7% to 30.4% between audit periods. Two companies that had not been named in the original study, Amazon and Kairos, showed overall PPB error rates of 8.66% and 6.60% and darker-female error rates of 31.37% and 22.50% [9].

Government testing reached the same conclusion at much larger scale. NISTIR 8280, "Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects," by Patrick Grother, Mei Ngan, and Kayee Hanaoka, published in December 2019, ran 189 mostly commercial algorithms from 99 developers over 18.27 million images of 8.49 million people. Its main result was that "false positive differentials are much larger than those related to false negatives and exist broadly, across many, but not all, algorithms tested," with false positive rates varying "by factors of 10 to beyond 100 times" across demographics. False positives were highest in West and East African and East Asian people on application photographs and highest in American Indians on domestic law enforcement images, and were consistently higher in women than men. A number of algorithms developed in China reversed the pattern, returning low false positive rates on East Asian faces [10].

### Google Photos and the gorilla label

On 29 June 2015 the programmer Jacky Alciné posted that [Google](https://aiwiki.ai/wiki/google) Photos had labeled photographs of him and a friend, both Black, as "gorillas." Yonatan Zunger, then Google's chief architect of social, responded publicly, and a company spokesperson said "We're appalled and genuinely sorry that this happened. We are taking immediate action to prevent this type of result from appearing" [11].

The remedy turned out to be suppression rather than repair. Reporting in January 2018 found that Google Photos had removed gorilla, chimp, and chimpanzee from the labels it would return, while other Google [computer vision](https://aiwiki.ai/wiki/computer_vision) services still recognized them [12]. A New York Times test in May 2023 searched a set of 44 images in both Google Photos and Apple Photos and found that neither returned results for most primates; a Google spokesman said the benefit of the label "does not outweigh the risk of harm," and Apple declined to comment [13]. Eight years after the incident, neither company had shown that the classifier was fixed; both had withdrawn the label instead.

### Health care and welfare risk scoring

Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan published "Dissecting racial bias in an algorithm used to manage the health of populations" in Science on 25 October 2019. The commercial algorithm they studied ranked patients for extra care management. It used predicted health care costs as the target. Because less money is historically spent on Black patients at any given level of illness, Black patients at a given risk score were considerably sicker than white patients at the same score. Correcting the disparity would have raised the share of Black patients flagged for additional help from 17.7% to 46.5% [14].

Amsterdam's Smart Check shows what happened when a government set out to avoid this class of failure deliberately. Development began in 2019. The model was an explainable boosting machine using 15 features measuring applicant behavior and characteristics, with gender, nationality, age, and postal code deliberately excluded. Early testing showed it was almost twice as likely to wrongly flag applicants of non-Western nationality, so the city reweighted the training data to equalize false-flag rates. In the 2023 live pilot, covering roughly 1,600 applications, the bias reversed: the reweighted model wrongly flagged Dutch nationals and women more often than the groups it had been corrected for, with one analysis putting women at 22 percent more likely to be wrongly flagged than men. The city halted the project in November 2023. MIT Technology Review, Lighthouse Reports, and the Dutch newspaper Trouw published the reconstruction in June 2025 [15][16].

## Formal definitions of fairness

The research field responded to these cases by writing down what "fair" should mean, and quickly found that the candidate definitions conflict.

| Criterion | Requirement | Standard reference |
| --- | --- | --- |
| [Demographic parity](https://aiwiki.ai/wiki/demographic_parity) (statistical parity) | The demographics of those receiving a given classification match the demographics of the population | Dwork, Hardt, Pitassi, Reingold and Zemel, 2011 [17] |
| [Equalized odds](https://aiwiki.ai/wiki/equalized_odds) | True positive and false positive rates are equal across groups | Hardt, Price and Srebro, 2016 [18] |
| Equal opportunity | True positive rates are equal across groups, a relaxation of equalized odds | Hardt, Price and Srebro, 2016 [18] |
| Calibration within groups | For each group and each score bin, the expected fraction of positives equals the bin's score | Kleinberg, Mullainathan and Raghavan, 2016 [19] |
| Balance for the positive class | The average score assigned to true positives is equal across groups | Kleinberg, Mullainathan and Raghavan, 2016 [19] |
| Balance for the negative class | The average score assigned to true negatives is equal across groups | Kleinberg, Mullainathan and Raghavan, 2016 [19] |
| [Individual fairness](https://aiwiki.ai/wiki/individual_fairness) | Similar individuals, under a task-specific similarity metric, are treated similarly | Dwork, Hardt, Pitassi, Reingold and Zemel, 2011 [17] |

The 2011 Dwork paper sets the individual and group views explicitly against each other. Its Lipschitz condition requires that two individuals who are close under a task-specific similarity metric receive similarly close output distributions, which pushes the hard question into the choice of metric; the same paper presents statistical parity separately, as a mechanism for "fair affirmative action" rather than as the default target [17]. Hardt, Price, and Srebro's 2016 method is different in kind: it is a post-processing step that adjusts an already-trained predictor, and the authors argue it "improves incentives by shifting the cost of poor classification from disadvantaged groups to the decision maker" [18]. Later work proposed further criteria, among them [counterfactual fairness](https://aiwiki.ai/wiki/counterfactual_fairness), under which "a decision is fair towards an individual if it is the same in (a) the actual world and (b) a counterfactual world where the individual belonged to a different demographic group" [41].

## The impossibility results

Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan posted "Inherent Trade-Offs in the Fair Determination of Risk Scores" on 19 September 2016, four months after the COMPAS story. They formalize calibration within groups, balance for the negative class, and balance for the positive class, then prove Theorem 1.1: any instance admitting a risk assignment that satisfies all three "must either allow for perfect prediction ... or have equal base rates." Approximate satisfaction of the three conditions forces the instance to approximately look like one of those two cases. The result is about risk assignments in general, so it "applies independently of the method" used to construct them, "algorithmic or otherwise," and the authors note that it holds equally "whether the risk score is determined by an algorithm or by a system of human decision-makers" [19].

Alexandra Chouldechova reached a closely related result from the recidivism side, posted on 28 February 2017. Her paper shows that the fairness criteria at issue "cannot all be simultaneously satisfied when recidivism prevalence differs across groups," and traces how disparate impact then follows from an instrument that fails error rate balance [20].

Between them the two papers resolve the ProPublica and Northpointe exchange. Northpointe was right that COMPAS satisfied predictive parity, ProPublica was right that it violated error rate balance, and because rearrest base rates differed between the groups no tool could have satisfied both. The practical reading of this [incompatibility of fairness metrics](https://aiwiki.ai/wiki/incompatibility_of_fairness_metrics) is that deploying a risk score requires choosing which fairness property to give up, and that the choice is a policy decision rather than a modeling one.

## Detection, auditing, and measurement

Because aggregate accuracy hides subgroup failure, the standard first move is disaggregated evaluation: report every metric separately by group and by intersection of groups. Gender Shades is the canonical demonstration: overall accuracy across the whole benchmark was 93.7 percent for Microsoft, 90.0 percent for Face++, and 87.9 percent for IBM, numbers that reveal nothing about the 20.8 to 34.7 percent error on darker-skinned women [8].

Documentation practices followed. "Model Cards for Model Reporting," by Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru, proposed short documents released with a trained model giving "benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups," together with intended use and known limitations [22]. The [model card](https://aiwiki.ai/wiki/model_card) is now a common artifact on model release, though its contents are not standardized.

Open-source tooling arrived around the same time. [AI Fairness 360](https://aiwiki.ai/wiki/aif360), described by Rachel Bellamy and colleagues in October 2018, packages fairness metrics for datasets and models, explanations of those metrics, and mitigation algorithms [23]. [Fairlearn](https://aiwiki.ai/wiki/fairlearn), an open-source, community-driven project, provides a comparable Python library plus educational material, and its documentation is unusually direct about what tooling cannot do. Because the sources of unfairness are partly societal and partly technical, it says, "it is not possible to fully 'debias' a system or to guarantee fairness." The stated goal is instead to let people "assess fairness-related harms, review the impacts of different mitigation strategies and then make trade-offs appropriate to their scenario" [24].

Independent auditing sits above the tooling. NIST's FRVT program is the most established example of a repeated quantitative evaluation across an entire vendor market, and its reports note that "reporting of demographic effects often has been incomplete in academic papers and in media coverage," particularly where only false negatives are discussed [10]. NIST followed SP 1270 with the [AI Risk Management Framework](https://aiwiki.ai/wiki/nist_ai_rmf), released on 26 January 2023 and intended for voluntary use in incorporating trustworthiness considerations into the design, development, use, and evaluation of AI systems [43].

## Mitigation and its limits

Technical interventions divide into three stages by where they act.

| Stage | Approach | Examples |
| --- | --- | --- |
| Pre-processing | Change the data before training | Reweighting or resampling training examples, collecting more representative data, relabeling |
| In-processing | Change the objective or the training procedure | Adding a fairness constraint or penalty to the loss, adversarial training against a protected attribute |
| Post-processing | Change the outputs of a trained model | Group-specific decision thresholds, as in the equalized odds derivation of Hardt, Price and Srebro [18] |

Each stage has costs. Post-processing with group-specific thresholds treats people differently according to a protected attribute, and some law forbids that outright: in United States employment testing, Title VII as amended in 1991 makes it unlawful "to adjust the scores of, use different cutoff scores for, or otherwise alter the results of, employment related tests on the basis of race, color, religion, sex, or national origin" [42]. In-processing requires the protected attribute at training time, which privacy rules may forbid collecting. Pre-processing is the least legally awkward and the least reliable, as Amsterdam's reweighting showed when the corrected model simply moved the harm onto different groups [15][16].

The harder limit is the impossibility results. Any mitigation optimizes a chosen metric, and moving one metric moves another. That is why current guidance treats bias as something to be measured, disclosed, and traded off in the open rather than eliminated, and why [responsible AI](https://aiwiki.ai/wiki/responsible_ai) programs pair technical measurement with governance, [explainable AI](https://aiwiki.ai/wiki/explainable_ai) work, appeal routes, and human review.

## Bias in language models and generative systems

Aylin Caliskan, Joanna Bryson, and Arvind Narayanan showed in work published in Science in 2017 that ordinary text corpora encode measurable human prejudice. Their Word Embedding Association Test (WEAT) replicated results from implicit-association psychology on GloVe vectors trained on web text, finding that language carries "recoverable and accurate imprints of our historic biases" [25]. Earlier work by Tolga Bolukbasi and colleagues, summarized in the Gender Shades paper, trained an analogy generator on [Word2Vec](https://aiwiki.ai/wiki/word2vec) and found that "man is to computer programmer as woman is to X" completed with "homemaker." Biases in an embedding of this kind are inherited by every system built on it, which is why [word embeddings](https://aiwiki.ai/wiki/word_embedding) became an early target for bias measurement [8].

Modern [large language models](https://aiwiki.ai/wiki/large_language_model) inherit the problem in less visible forms. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King reported in March 2024 that language models exhibit dialect prejudice: prompted with African American English, models produced covert stereotypes more negative than any experimentally recorded human stereotypes about African Americans, while their overt statements about African Americans were positive. In hypothetical decision tasks the models assigned speakers of the dialect to less prestigious jobs, convicted them more often, and sentenced them to death more often. Human feedback training reduced the overt bias without touching the covert one, and in some cases widened the gap [26].

[OpenAI](https://aiwiki.ai/wiki/openai) published its own counterfactual evaluation, "First-Person Fairness in Chatbots," in October 2024. Tyna Eloundou, Alex Beutel, and colleagues varied the user's name across millions of [ChatGPT](https://aiwiki.ai/wiki/chatgpt) interactions spanning six models, 66 tasks, and nine domains, using a language model to score harmful stereotypes in the responses. They describe it as the first large-scale fairness evaluation based on real-world chat data, and report that post-training reinforcement learning significantly reduced the measured biases [27]. Counterfactual probes of this kind now sit alongside [red teaming](https://aiwiki.ai/wiki/red_teaming) in the pre-release evaluation of [generative AI](https://aiwiki.ai/wiki/generative_ai) systems.

## Law and regulation

### United States

In the United States, bias claims run through existing civil rights law rather than a dedicated AI statute, principally the [disparate impact](https://aiwiki.ai/wiki/disparate_impact) doctrine, under which Title VII "prohibits employers from using neutral tests or selection procedures that have the effect of disproportionately excluding persons" on protected grounds unless the procedure is "job-related and consistent with business necessity," and under which a challenger may still prevail by identifying a less discriminatory alternative [28].

Two 2025 executive orders pulled in opposite directions. Executive Order 14281, "Restoring Equality of Opportunity and Meritocracy," signed 23 April 2025, declares it federal policy "to eliminate the use of disparate-impact liability in all contexts to the maximum degree possible" and directs all agencies to deprioritize enforcement of statutes and regulations that include it [37]. Executive Order 14319, "Preventing Woke AI in the Federal Government," signed 23 July 2025, directs federal agencies to procure large language models that comply with two "Unbiased AI Principles," truth-seeking and ideological neutrality, the latter defined as being "neutral, nonpartisan tools that do not manipulate responses in favor of ideological dogmas such as DEI" [38]. The two orders reflect competing definitions of what counts as bias in a model.

Private litigation has proceeded in parallel. In Mobley v. Workday, No. 3:23-cv-00770 in the Northern District of California, the court granted preliminary certification of a nationwide collective under the Age Discrimination in Employment Act on 16 May 2025, covering applicants aged 40 and over who applied for jobs through Workday's platform and were denied employment recommendations. The ruling followed an earlier holding that vendors of AI screening tools can be directly liable under federal anti-discrimination law when their tools act as gatekeepers in hiring [39].

State and city rules have moved faster than federal ones.

| Rule | Jurisdiction | Requirement | Status |
| --- | --- | --- | --- |
| Local Law 144 of 2021 | New York City | Annual independent bias audit of automated employment decision tools, publication of selection rate and impact ratio by sex, race and ethnicity, and intersectional categories, plus ten business days' notice to candidates | Enforcement began 5 July 2023 [34] |
| SB 24-205 | Colorado | Reasonable care by developers and deployers of high-risk systems to protect consumers from known or reasonably foreseeable risks of algorithmic discrimination, with a rebuttable presumption for disclosure and impact assessment compliance | Signed 17 May 2024, effective date postponed from 1 February 2026 to 30 June 2026 by SB 25B-004 (signed 28 August 2025) [35][40] |
| SB 26-189 | Colorado | Repeals and reenacts SB 24-205 with developer technical documentation duties, consumer notice at the point of interaction, adverse-outcome disclosure within 30 days, and rights to data correction and human reconsideration | Signed 14 May 2026, principal requirements from 1 January 2027 [36] |

### European Union

The [EU AI Act](https://aiwiki.ai/wiki/eu_ai_act) entered into force on 1 August 2024. Prohibitions and AI literacy duties applied from 2 February 2025 and general-purpose model rules from 2 August 2025 [31]. Article 5 bans social scoring that leads to detrimental treatment unrelated or disproportionate to the underlying behavior, and bans risk assessments predicting criminal offending "based solely on the profiling of a natural person or on assessing their personality traits and characteristics," while allowing systems that support a human assessment already grounded in objective, verifiable facts [30].

Article 10 is the bias-specific provision for high-risk systems. Paragraph 2 requires examination of datasets for "possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law," and for "appropriate measures to detect, prevent and mitigate possible biases identified." Paragraph 5 permits providers to process special categories of personal data specifically for bias detection and correction, subject to conditions including that no other data would work, strict access controls, no third-party sharing, and deletion once the correction is done, which is a deliberate carve-out from the general prohibition on processing sensitive data under the [GDPR](https://aiwiki.ai/wiki/gdpr) [29].

The high-risk deadlines then moved. The Commission proposed targeted amendments on 19 November 2025 in the [digital omnibus](https://aiwiki.ai/wiki/eu_ai_act_digital_omnibus). The Council agreed its general approach on 13 March 2026, the Parliament's Internal Market and Civil Liberties committees adopted a joint position on 18 March 2026 by 101 votes to 9 with 8 abstentions, and co-legislators reached a trilogue agreement on 7 May 2026. Parliament approved the result in plenary on 16 June 2026 by 423 votes to 57 with 174 abstentions. Stand-alone high-risk systems, including those used in biometrics, employment, education, essential services, law enforcement, and border management, now apply from 2 December 2027, and high-risk systems embedded in regulated products from 2 August 2028 [32][33].

## See also

- [Algorithmic fairness](https://aiwiki.ai/wiki/algorithmic_fairness)
- [AI ethics](https://aiwiki.ai/wiki/ai_ethics)
- [COMPAS](https://aiwiki.ai/wiki/compas)
- [Incompatibility of fairness metrics](https://aiwiki.ai/wiki/incompatibility_of_fairness_metrics)
- [Facial recognition](https://aiwiki.ai/wiki/facial_recognition)
- [EU AI Act](https://aiwiki.ai/wiki/eu_ai_act)

## References

1. Friedman, B. and Nissenbaum, H. "Bias in Computer Systems." ACM Transactions on Information Systems, Vol. 14, No. 3, July 1996, pp. 330-347. https://nissenbaum.tech.cornell.edu/papers/biasincomputers.pdf
2. Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A. and Hall, P. "Towards a Standard for Identifying and Managing Bias in Artificial Intelligence." NIST Special Publication 1270, March 2022. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf
3. Angwin, J., Larson, J., Mattu, S. and Kirchner, L. "Machine Bias." ProPublica, 23 May 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
4. Larson, J., Mattu, S., Kirchner, L. and Angwin, J. "How We Analyzed the COMPAS Recidivism Algorithm." ProPublica, 23 May 2016. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm
5. Dieterich, W., Mendoza, C. and Brennan, T. "COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity." Northpointe Inc. Research Department, 8 July 2016. https://njoselson.github.io/pdfs/ProPublica_Commentary_Final_070616.pdf
6. Larson, J. and Angwin, J. "Technical Response to Northpointe." ProPublica, 29 July 2016. https://www.propublica.org/article/technical-response-to-northpointe
7. "Amazon ditched AI recruitment software because it was biased against women." MIT Technology Review, 10 October 2018. https://www.technologyreview.com/2018/10/10/139858/amazon-ditched-ai-recruitment-software-because-it-was-biased-against-women/
8. Buolamwini, J. and Gebru, T. "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." Proceedings of Machine Learning Research, Vol. 81, Conference on Fairness, Accountability and Transparency, 2018, pp. 1-15. https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf
9. Raji, I.D. and Buolamwini, J. "Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products." AAAI/ACM Conference on AI, Ethics, and Society, 2019. https://www.aies-conference.com/2019/wp-content/uploads/2019/01/AIES-19_paper_223.pdf
10. Grother, P., Ngan, M. and Hanaoka, K. "Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects." NISTIR 8280, National Institute of Standards and Technology, December 2019. https://nvlpubs.nist.gov/nistpubs/ir/2019/NIST.IR.8280.pdf
11. "Google apologizes for mis-tagging photos of African Americans." CBS News, 1 July 2015. https://www.cbsnews.com/news/google-photos-labeled-pics-of-african-americans-as-gorillas/
12. "Google Photos Still Has a Problem with Gorillas." MIT Technology Review, 11 January 2018. https://www.technologyreview.com/2018/01/11/146257/google-photos-still-has-a-problem-with-gorillas/
13. "Google's Photos App is Still Unable to Find Gorillas." PetaPixel, 22 May 2023, reporting a New York Times test of Google Photos and Apple Photos. https://petapixel.com/2023/05/22/googles-photos-app-is-still-unable-to-find-gorillas/
14. Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. "Dissecting racial bias in an algorithm used to manage the health of populations." Science, Vol. 366, No. 6464, 25 October 2019, pp. 447-453. https://pubmed.ncbi.nlm.nih.gov/31649194/
15. Guo, E., Geiger, G. and Braun, J.-C. "Inside Amsterdam's high-stakes experiment to create fair welfare AI." MIT Technology Review, 11 June 2025. https://www.technologyreview.com/2025/06/11/1118233/amsterdam-fair-welfare-ai-discriminatory-algorithms-failure/
16. "How we investigated Amsterdam's attempt to build a 'fair' fraud detection model." Lighthouse Reports methodology note. https://www.lighthousereports.com/methodology/amsterdam-fairness/
17. Dwork, C., Hardt, M., Pitassi, T., Reingold, O. and Zemel, R. "Fairness Through Awareness." arXiv:1104.3913, 20 April 2011. https://arxiv.org/abs/1104.3913
18. Hardt, M., Price, E. and Srebro, N. "Equality of Opportunity in Supervised Learning." arXiv:1610.02413, 7 October 2016. https://arxiv.org/abs/1610.02413
19. Kleinberg, J., Mullainathan, S. and Raghavan, M. "Inherent Trade-Offs in the Fair Determination of Risk Scores." arXiv:1609.05807, 19 September 2016. https://arxiv.org/abs/1609.05807
20. Chouldechova, A. "Fair prediction with disparate impact: A study of bias in recidivism prediction instruments." arXiv:1703.00056, 28 February 2017. https://arxiv.org/abs/1703.00056
21. Ensign, D., Friedler, S.A., Neville, S., Scheidegger, C. and Venkatasubramanian, S. "Runaway Feedback Loops in Predictive Policing." arXiv:1706.09847, 29 June 2017. https://arxiv.org/abs/1706.09847
22. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D. and Gebru, T. "Model Cards for Model Reporting." arXiv:1810.03993, 5 October 2018. https://arxiv.org/abs/1810.03993
23. Bellamy, R.K.E. et al. "AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias." arXiv:1810.01943, 3 October 2018. https://arxiv.org/abs/1810.01943
24. "About Fairlearn." Fairlearn project documentation. https://fairlearn.org/main/about/index.html
25. Caliskan, A., Bryson, J.J. and Narayanan, A. "Semantics derived automatically from language corpora contain human-like biases." Science, 2017; preprint arXiv:1608.07187. https://arxiv.org/abs/1608.07187
26. Hofmann, V., Kalluri, P.R., Jurafsky, D. and King, S. "Dialect prejudice predicts AI decisions about people's character, employability, and criminality." arXiv:2403.00742, 1 March 2024. https://arxiv.org/abs/2403.00742
27. Eloundou, T., Beutel, A., Robinson, D.G., Gu-Lemberg, K., Brakman, A.-L., Mishkin, P., Shah, M., Heidecke, J., Weng, L. and Kalai, A.T. "First-Person Fairness in Chatbots." arXiv:2410.19803, 16 October 2024. https://arxiv.org/abs/2410.19803
28. "Employment Tests and Selection Procedures." U.S. Equal Employment Opportunity Commission. https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures
29. EU Artificial Intelligence Act, Article 10: Data and Data Governance. https://artificialintelligenceact.eu/article/10/
30. EU Artificial Intelligence Act, Article 5: Prohibited AI Practices. https://artificialintelligenceact.eu/article/5/
31. "AI Act Implementation Timeline." EU Artificial Intelligence Act. https://artificialintelligenceact.eu/implementation-timeline/
32. "Digital omnibus on AI." European Parliament Legislative Train Schedule, updated 20 June 2026. https://www.europarl.europa.eu/legislative-train/package-digital-package/file-digital-omnibus-on-ai
33. "MEPs support postponement of certain rules on artificial intelligence." European Parliament press release, 18 March 2026. https://www.europarl.europa.eu/news/en/press-room/20260316IPR38219/meps-support-postponement-of-certain-rules-on-artificial-intelligence
34. "New York City Begins Enforcing Restrictions on the Use of Artificial Intelligence." Norton Rose Fulbright. https://www.nortonrosefulbright.com/en/knowledge/publications/e90da0c0/new-york-city-begins-enforcing-restrictions-on-the-use-of-artificial-intelligence
35. Colorado Senate Bill 24-205, "Consumer Protections for Artificial Intelligence." Colorado General Assembly. https://leg.colorado.gov/bills/sb24-205
36. Colorado Senate Bill 26-189, "Automated Decision-Making Technology." Colorado General Assembly. https://leg.colorado.gov/bills/sb26-189
37. Executive Order 14281, "Restoring Equality of Opportunity and Meritocracy," 23 April 2025. https://www.whitehouse.gov/presidential-actions/2025/04/restoring-equality-of-opportunity-and-meritocracy/
38. Executive Order 14319, "Preventing Woke AI in the Federal Government," 23 July 2025. https://www.whitehouse.gov/presidential-actions/2025/07/preventing-woke-ai-in-the-federal-government/
39. "AI Screening Tools Under Scrutiny: Federal Court Preliminarily Certifies Nationwide Collective in AI Hiring Age Discrimination Case." Davis Wright Tremaine, May 2025. https://www.dwt.com/blogs/employment-labor-and-benefits/2025/05/ai-hiring-age-discrimination-federal-court-workday
40. Colorado Senate Bill 25B-004, "Increase Transparency for Algorithmic Systems." Colorado General Assembly. https://leg.colorado.gov/bills/sb25b-004
41. Kusner, M.J., Loftus, J.R., Russell, C. and Silva, R. "Counterfactual Fairness." arXiv:1703.06856, 20 March 2017. https://arxiv.org/abs/1703.06856
42. 42 U.S.C. 2000e-2(l), "Prohibition of discriminatory use of test scores." Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/uscode/text/42/2000e-2
43. "AI Risk Management Framework." National Institute of Standards and Technology. AI RMF 1.0 released 26 January 2023. https://www.nist.gov/itl/ai-risk-management-framework