Machine learning terms/Fairness

See also: Machine learning terms

The key machine learning fairness terms are the formal criteria used to define and measure when a model treats demographic groups equitably, together with the named biases that make models unfair. The most cited fairness criteria are demographic parity, equalized odds, equality of opportunity, predictive parity, calibration, individual fairness, and counterfactual fairness; the most cited bias concepts include sampling bias, label bias, measurement bias, disparate impact, disparate treatment, fairness through unawareness, group attribution bias, and in-group bias ^[1][2][3]. A foundational result of the field, the fairness impossibility theorem, proves that several of these criteria (calibration, balanced false positive rates, and balanced false negative rates) cannot all hold at once when base rates differ across groups, so choosing a fairness definition is a value judgment rather than a purely technical decision ^[4][5].

In the taxonomy used by Google's machine learning glossary, demographic parity "is satisfied if the results of a model's classification are not dependent on a given sensitive attribute," equalized odds checks "whether a model is predicting outcomes equally well for all values of a sensitive attribute with respect to both the positive class and negative class," and group attribution bias is the error of "assuming that what is true for an individual is also true for everyone in that group" ^[1].

introduction

Machine learning fairness is the subfield of machine learning and artificial intelligence ethics concerned with detecting, measuring, and mitigating unjust outcomes that ML systems produce for individuals or groups defined by protected attributes such as race, sex, age, disability, religion, or national origin. The field draws on statistics, computer science, law, and moral philosophy, and has grown rapidly since the early 2010s as automated decision systems have moved into hiring, lending, criminal justice, healthcare, education, advertising, and content moderation.

Fairness matters for several reasons. Models trained on historical data can reproduce or amplify human discrimination encoded in that data, disadvantaging already marginalized groups. Because the same model is applied at scale, a small disparity in error rate can translate into thousands of harmful decisions. Fairness is also a legal requirement in many jurisdictions: statutes such as the U.S. Civil Rights Act of 1964 (Title VII), the Fair Housing Act, the Equal Credit Opportunity Act, and the U.K. Equality Act 2010 apply to automated decisions just as to human ones.

The technical literature distinguishes related notions. Bias in the statistical sense is systematic error of a model relative to ground truth. Bias in the social sense, sometimes called bias (ethics/fairness), is systematic difference in outcomes across demographic groups, and this kind of algorithmic bias is what fairness work primarily targets. Discrimination in the legal sense is differential treatment or impact on members of a protected class. A system can be statistically unbiased yet socially biased, or socially fair yet legally non-compliant, so practitioners must be precise about which definition is in play.

What are the main types of bias in machine learning?

Machine learning bias arises at every stage of the development and deployment pipeline. The list below groups common sources by where they enter the system.

data bias

Data bias is bias that originates in the dataset used to train a model. The training set may not represent the population the model will be deployed on, the labels may be unreliable, or the features may measure different constructs across groups.

Scroll sideways for more →

algorithmic bias

Algorithmic bias is bias introduced by the choice of model family, loss function, feature engineering, or optimization procedure, even when the data itself is balanced. Examples include models that minimize average loss and therefore perform worse on minority subgroups, regularizers that disproportionately shrink coefficients on rare features, and feature pipelines that drop sparse signals important to a smaller subgroup.

deployment and feedback bias

Deployment bias appears after the model is in production. A model used differently from how it was designed, used in a different population than its training set, or used in a feedback loop where its own predictions influence future training data can produce unfair outcomes even if it was fair on the original test set. Automation bias is the human tendency to trust machine outputs even when they are wrong, which can compound model bias by removing the human check.

cognitive bias entering ML

Human cognitive biases shape labeling, model design, and evaluation. Google's glossary describes group attribution bias as "assuming that what is true for an individual is also true for everyone in that group," and in-group bias as "showing partiality to one's own group or own characteristics" ^[1]. Other recognized examples include confirmation bias, implicit bias, out-group homogeneity bias, and experimenter's bias. These biases can be embedded into a system through choices about which features to collect, which labels to apply, and how to interpret model errors.

notable case studies

A handful of high-profile audits drove fairness research from a niche academic concern into a mainstream engineering priority. Each case study illustrates how a different stage of the pipeline can produce disparate outcomes.

Scroll sideways for more →

These cases share a common pattern: a model deployed at scale, a sensitive demographic characteristic correlated with the prediction target through a flawed proxy, and a downstream harm that disproportionately affected one group.

What are the main fairness criteria in machine learning?

Fairness researchers have proposed dozens of formal criteria. The most widely cited are listed below. Notation: let A be a sensitive attribute (such as race), Y the true outcome, and Ŷ the model's prediction. P denotes probability. Most of these criteria fall into three statistical families: independence (demographic parity), separation (equalized odds, equality of opportunity), and sufficiency (predictive parity, calibration), a grouping introduced in Fairness and Machine Learning by Barocas, Hardt, and Narayanan ^[3].

Scroll sideways for more →

What is demographic parity?

Demographic parity, also called statistical parity or independence, holds when the positive prediction rate is identical across groups, regardless of the true outcome. In Google's framing it "is satisfied if the results of a model's classification are not dependent on a given sensitive attribute" ^[1]. It is the criterion most often required for advertising and outreach, where the goal is equal representation, but it can force a model to ignore genuine differences in base rates, which is why critics note it can require accepting less-qualified candidates from one group to equalize rates.

What is equalized odds?

Equalized odds requires that both the true positive rate and the false positive rate are equal across groups: qualified members of every group have the same chance of a positive decision, and unqualified members have the same chance of a negative one. Google defines it as a metric for "whether a model is predicting outcomes equally well for all values of a sensitive attribute with respect to both the positive class and negative class" ^[1]. It was formalized by Hardt, Price, and Srebro in their 2016 paper "Equality of Opportunity in Supervised Learning," who describe their criterion as "oblivious," depending "only on the joint statistics of the predictor, the target and the protected attribute, but not on interpretation (features) of individual examples" ^[11].

What is equality of opportunity?

Equality of opportunity is a relaxation of equalized odds that requires only the true positive rate to be equal across groups, that is, among people who genuinely deserve the positive outcome (Y = 1), the model identifies them at equal rates ^[11]. Google describes it as a metric for "whether a model is predicting the desirable outcome equally well for all values of a sensitive attribute" ^[1]. It is often preferred when false negatives (missing a deserving person) are the harm of concern, such as denying a loan to a creditworthy applicant.

What is predictive parity?

Predictive parity holds when the positive predictive value (precision) is equal across groups: when the model issues a positive prediction, it is correct at the same rate for every group. Google describes it as checking "whether, for a given classification model, the precision rates are equivalent for subgroups under consideration" ^[1]. Northpointe (now Equivant) defended the COMPAS tool on grounds close to predictive parity and calibration, which is precisely the criterion that the impossibility theorem shows cannot coexist with equalized error rates when base rates differ ^[4][6].

Can all fairness criteria be satisfied at once?

No. A central result of the field is that the most popular fairness criteria are mutually incompatible except in trivial cases (equal base rates across groups, or a perfect predictor). Three formulations of this result appeared close together.

• Chouldechova (2017), in "Fair prediction with disparate impact: A study of bias in recidivism prediction instruments," proved that for a binary classifier with non-equal base rates between groups, calibration (predictive parity) and equal false positive and false negative rates cannot all hold simultaneously ^[4]. The result directly applies to the COMPAS controversy: Northpointe (now Equivant) argued the tool was calibrated, ProPublica argued it had unequal error rates, and Chouldechova showed both could be true and the two cannot be jointly satisfied when base rates differ.
• Kleinberg, Mullainathan, and Raghavan (2016), in "Inherent trade-offs in the fair determination of risk scores," independently proved a stronger form: for non-equal base rates, it is impossible for a probabilistic risk score to be calibrated within groups and also satisfy balance for the positive class and balance for the negative class ^[5]. They write that "these three conditions cannot in general be simultaneously satisfied except in highly constrained special cases."
• Pleiss, Raghavan, Wu, Kleinberg, and Weinberger (2017) in "On Fairness and Calibration," extended this to show that calibration is incompatible with equalized odds even when one allows randomized post-processing ^[12].

This cluster of results is sometimes called the incompatibility of fairness metrics or simply the impossibility theorem. It does not mean fairness is unattainable; it means choosing a definition is a value judgment that cannot be deferred to mathematics. Practitioners must decide which kind of fairness is appropriate for the application at hand.

How is bias mitigated in machine learning?

Mitigation techniques fall into three families based on where in the pipeline they intervene. Most production systems combine techniques from more than one family.

pre-processing

Pre-processing methods modify the training data before model fitting. Examples:

• Reweighting (Kamiran and Calders, 2012): assign higher sample weights to under-represented combinations of label and protected attribute so the weighted distribution satisfies demographic parity.
• Disparate impact remover (Feldman et al., 2015): repair feature distributions so that conditional distributions match across groups while preserving within-group rank order ^[13].
• Learning fair representations (Zemel et al., 2013): learn an intermediate representation informative about the label but uninformative about the protected attribute.
• SMOTE (Chawla et al., 2002): generate synthetic examples for under-represented groups.
• Causal pre-processing: adjust features that are descendants of the sensitive attribute on a causal model.

The term preprocessing covers these techniques. They are model-agnostic but can degrade overall accuracy.

in-processing

In-processing methods change the training procedure, typically by adding fairness terms to the loss function or imposing constraints on the optimization.

• Constrained optimization (Zafar et al., 2017): minimize loss subject to a covariance-style constraint approximating demographic parity or equal opportunity ^[18].
• Adversarial debiasing (Zhang, Lemoine, Mitchell, 2018): train the predictor jointly with an adversary that tries to recover the protected attribute from the prediction. The predictor is rewarded for fooling the adversary, removing protected-attribute information from its outputs ^[19].
• Reductions approach (Agarwal et al., 2018): reduce fair classification to a sequence of cost-sensitive classification problems converging to a model satisfying a chosen group fairness criterion. Implemented in Fairlearn via exponentiated-gradient and grid-search.
• Distributionally robust optimization (Hashimoto et al., 2018): minimize worst-case loss over sub-populations, improving minority-group performance without requiring protected-attribute labels.

post-processing

Post-processing methods adjust the outputs of an already-trained model. They are useful when retraining is expensive or the model is supplied by a third party. The umbrella term is post-processing.

• Threshold adjustment: choose a different decision threshold for each group so that, for example, true positive rates are equalized. Hardt, Price, and Srebro (2016) gave the canonical algorithm for satisfying equalized odds by post-processing ^[11].
• Calibration by group: re-calibrate predicted probabilities separately within each group using isotonic regression or Platt scaling.
• Reject option classification (Kamiran, Karim, Zhang, 2012): for examples near the decision boundary, override the prediction in favor of the disadvantaged group.

What tools and libraries measure ML fairness?

A growing ecosystem of open-source libraries supports fairness measurement and mitigation. The most widely used are listed below.

Scroll sideways for more →

Larger MLOps platforms (Amazon SageMaker Clarify, Google Vertex AI Model Monitoring, Azure Responsible AI dashboard) bundle subsets of these capabilities with the rest of the model lifecycle.

How is ML fairness regulated?

Fairness is increasingly a matter of statute, not only voluntary best practice. The most consequential current rules are summarized below.

Scroll sideways for more →

In addition, U.S. financial regulators (CFPB, OCC, Federal Reserve) have issued guidance reaffirming that the Equal Credit Opportunity Act and Regulation B apply to algorithmic underwriting and that creditors must provide specific reasons for adverse actions, including those produced by complex models.

Who are notable researchers in ML fairness?

A short, non-exhaustive list of researchers whose work has shaped the field:

Scroll sideways for more →

The annual ACM Conference on Fairness, Accountability, and Transparency (FAccT, formerly FAT* and FAT/ML) launched in 2018 and serves as the field's primary publication venue.

How do you run a fairness workflow?

A typical fairness workflow combines several activities:

1. Define scope. Identify protected groups, the relevant legal regime, and possible harms. Choose a fairness criterion: equalized odds for risk-screening, calibration for resource allocation, demographic parity for advertising.
2. Audit the data. Compute group-level statistics for the training set and validation set. Look for missing groups, label-quality differences, and proxy features.
3. Audit the model. Compute the chosen fairness metrics on the test set with confidence intervals, since group sizes often vary.
4. Mitigate. Apply pre-, in-, or post-processing as appropriate. Document the trade-offs.
5. Deploy with monitoring. Compute fairness metrics on production traffic and trigger alerts when they drift. Maintain a feedback channel for affected users.
6. Document. Publish a model card or datasheet covering intended use, evaluation by group, and known limitations.

ELI5: fairness in machine learning

Imagine a robot deciding who gets into a club. Demographic parity says the robot should let in the same fraction of tall people as short people. Equal opportunity says that, among people who actually belong in the club, the robot should be just as good at recognizing the tall ones as the short ones. The hard part, proven by mathematicians, is that you usually cannot make the robot fair in every way at the same time, so the people who build it have to choose which kind of fairness matters most for that club. That choice is about values, not just math.

index of ML fairness term wiki pages

The following pages on this wiki cover individual fairness concepts in detail.

• attribute
• automation bias
• bias (ethics/fairness)
• confirmation bias
• counterfactual fairness
• coverage bias
• demographic parity
• disparate impact
• disparate treatment
• equality of opportunity
• equalized odds
• experimenter's bias
• fairness constraint
• fairness metric
• group attribution bias
• implicit bias
• incompatibility of fairness metrics
• individual fairness
• in-group bias
• non-response bias
• out-group homogeneity bias
• participation bias
• post-processing
• predictive parity
• predictive rate parity
• preprocessing
• proxy (sensitive attributes)
• reporting bias
• sampling bias
• selection bias
• sensitive attribute
• unawareness (to a sensitive attribute)

references

1. Google. "Machine Learning Glossary: Fairness (Responsible AI)." Google for Developers. developers.google.com/machine-learning/glossary/fairness. ↩
2. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A. (2021). "A Survey on Bias and Fairness in Machine Learning." *ACM Computing Surveys* 54(6):1 to 35. ↩
3. Barocas, S., Hardt, M., Narayanan, A. (2023). *Fairness and Machine Learning: Limitations and Opportunities*. MIT Press. fairmlbook.org. ↩
4. Chouldechova, A. (2017). "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." *Big Data* 5(2):153 to 163. ↩
5. Kleinberg, J., Mullainathan, S., Raghavan, M. (2016). "Inherent Trade-Offs in the Fair Determination of Risk Scores." arXiv:1609.05807; ITCS 2017. ↩
6. Angwin, J., Larson, J., Mattu, S., Kirchner, L. (2016). "Machine Bias." *ProPublica*, 23 May 2016. ↩
7. Dastin, J. (2018). "Amazon scraps secret AI recruiting tool that showed bias against women." *Reuters*, 10 October 2018. ↩
8. Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S. (2019). "Dissecting racial bias in an algorithm used to manage the health of populations." *Science* 366(6464):447 to 453. ↩
9. Buolamwini, J., Gebru, T. (2018). "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification." PMLR 81 (FAT*). ↩
10. Vigdor, N. (2019). "Apple Card Investigated After Gender Discrimination Complaints." *NYT*, 10 November 2019; NYDFS Report, 23 March 2021. ↩
11. Hardt, M., Price, E., Srebro, N. (2016). "Equality of Opportunity in Supervised Learning." NeurIPS (NIPS 2016), 3315 to 3323. ↩
12. Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., Weinberger, K. Q. (2017). "On Fairness and Calibration." NeurIPS. ↩
13. Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., Venkatasubramanian, S. (2015). "Certifying and Removing Disparate Impact." KDD. ↩
14. European Parliament and Council. (2024). Regulation (EU) 2024/1689 (EU AI Act). Official Journal, 12 July 2024. ↩
15. NIST. (2023). *AI Risk Management Framework (AI RMF 1.0)*. NIST AI 100-1. ↩
16. NYC DCWP. (2023). "Automated Employment Decision Tools (Local Law 144)." Effective 5 July 2023. ↩
17. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R. (2012). "Fairness Through Awareness." ITCS.
18. Zafar, M. B., Valera, I., Rodriguez, M. G., Gummadi, K. P. (2017). "Fairness Beyond Disparate Treatment and Disparate Impact." WWW. ↩
19. Zhang, B. H., Lemoine, B., Mitchell, M. (2018). "Mitigating Unwanted Biases with Adversarial Learning." AIES. ↩
20. Colorado General Assembly. (2024). SB 24-205, Consumer Protections in Interactions with AI Systems. ↩