Proxy (sensitive attributes)

A proxy for a sensitive attribute is an ordinary input feature that is statistically correlated with a protected characteristic (such as race, gender, age, religion, or disability) and therefore leaks information about that characteristic into a model, even when the protected attribute itself is never supplied as input. Common proxies include ZIP code (which correlates with race because of historical residential segregation), a person's name (which correlates with race, gender, and national origin), the school attended, browsing device, and predicted healthcare cost. Because supervised learners absorb any signal that improves predictive accuracy, a model trained on data that contains proxies can reproduce or amplify the same discrimination that the omitted attribute would have produced if it had been used directly ^[30]. The existence of proxies is the central reason that simply deleting a sensitive column from a training set, an approach known as fairness through unawareness, does not by itself prevent biased outcomes ^[2][30].

The study of proxy variables sits at the intersection of statistics, civil rights law, and algorithmic fairness. It became a public concern after a series of investigations into automated systems used in hiring, lending, criminal justice, and healthcare showed that ostensibly neutral models were producing systematically different outcomes for legally protected groups ^[8][18][23]. Researchers and regulators have responded with formal definitions of fairness, statistical tests for proxy detection, mitigation algorithms at every stage of the modeling pipeline, and documentation standards for datasets and models.

What is a proxy for a sensitive attribute?

A proxy is any observed variable X that carries information about a sensitive attribute S, in the sense that the conditional distribution P(S | X) differs meaningfully from the marginal distribution P(S). Equivalent ways of saying the same thing include: X has nonzero mutual information with S, X is predictive of S above the base rate, or X is statistically dependent on S. Proxies vary in strength. A near-perfect proxy lets a model reconstruct S almost exactly from X. A weak proxy carries only a small amount of information but, combined with other weak proxies, can still allow accurate reconstruction. Modern high-dimensional data, including text, images, click logs, and graph features, almost always contains weak proxies for common sensitive attributes ^[30].

The distinction between a sensitive attribute and a proxy is partly legal and partly statistical. The sensitive attribute is the characteristic that anti-discrimination law protects. The proxy is a feature that the model treats as ordinary input but that effectively encodes the protected characteristic. A model can engage in disparate treatment by acting on a proxy in the same way it would act on the protected attribute, and it can produce disparate impact by relying on a proxy whose distribution differs across protected groups, regardless of intent.

Which attributes count as sensitive or protected?

Which attributes count as sensitive depends on jurisdiction and context. The relevant categories under United States federal law include the following.

Scroll sideways for more →

The European Union's General Data Protection Regulation (GDPR) defines special categories of personal data in Article 9, including racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, genetic data, biometric data used for unique identification, health data, and data concerning sex life or sexual orientation. Processing these categories is generally prohibited unless one of the exceptions in Article 9(2) applies. The EU AI Act, which entered into force in 2024, classifies AI systems used in employment, credit scoring, education admissions, law enforcement, and migration as high-risk and requires providers to examine training, validation, and testing data sets for possible bias, and to conduct a fundamental rights impact assessment before deployment ^[28].

Many other regimes overlap with these. Some U.S. states protect additional categories such as sexual orientation, gender identity, source of income, or familial status. Brazil's LGPD, Canada's PIPEDA, and the United Kingdom's Equality Act 2010 each define their own lists. The practical effect is that what counts as a proxy depends on what the relevant law treats as sensitive in the specific use case.

What are common proxy variables?

A short catalog of proxies that recur in the empirical literature.

Scroll sideways for more →

ZIP code as a proxy for race is the canonical example in the U.S. context. Beginning in the 1930s, the Home Owners' Loan Corporation produced "residential security" maps that color-coded neighborhoods according to perceived investment risk; predominantly Black neighborhoods were outlined in red and effectively cut off from federally backed mortgage credit. Richard Rothstein's 2017 book The Color of Law documents how federal, state, and local governments enforced residential segregation throughout the twentieth century ^[15]. The result is that ZIP codes in many U.S. metropolitan areas remain strongly correlated with race decades after explicit racial covenants were outlawed by the Fair Housing Act, and any model that uses ZIP code as a feature is, in effect, partly using race.

Bertrand and Mullainathan's 2004 field experiment Are Emily and Greg More Employable than Lakisha and Jamal? demonstrated the same dynamic for given names ^[1]. The authors sent roughly 5,000 fictitious resumes to more than 1,300 job ads in Boston and Chicago, randomly assigning either stereotypically white names (such as Emily Walsh or Brendan Baker) or stereotypically Black names (such as Lakisha Washington or Jamal Jones). Resumes with white-sounding names received about 50 percent more callbacks for interviews, and the return to higher resume quality was larger for white-named applicants ^[1]. The racial gap was uniform across occupation, industry, and employer size. Names function as a strong proxy for race in this setting, which is why automated resume screeners that read names or name-derived features can reproduce the same disparity.

How do organizations infer a sensitive attribute they are not allowed to collect?

The same proxy structure that creates fairness problems is also used deliberately to estimate missing demographic data for audits. Under the Equal Credit Opportunity Act, non-mortgage lenders are generally prohibited from collecting an applicant's race or ethnicity, so the U.S. Consumer Financial Protection Bureau (CFPB) infers it for fair-lending analysis using Bayesian Improved Surname Geocoding (BISG). BISG first builds a probability of belonging to each race or ethnicity from the demographics associated with a person's surname, then updates that probability using the demographics of the Census block group of their address, combining two proxies (name and geography) through Bayes' rule ^[31]. The CFPB published the methodology in 2014, and it is now widely used by regulators and lenders to estimate disparities when the protected attribute is unavailable ^[31]. BISG illustrates the double edge of proxies: the same correlation that lets a screener accidentally discriminate also lets an auditor reconstruct the protected class well enough to measure that discrimination.

What are the most famous proxy discrimination cases?

Amazon hiring tool (2014 to 2017)

Reuters reported in October 2018 that Amazon had built an internal resume-screening model trained on a decade of hiring data, scrapped the project after engineers found that it systematically downgraded resumes that contained the word "women's" (as in "women's chess club captain" or "women's college"), and penalized graduates of two unnamed all-women's colleges ^[18]. The model had no explicit gender feature, but the historical training data reflected a male-dominated technical workforce, and the algorithm picked up gendered vocabulary as a proxy. Attempts to debias the lists of penalized terms could not guarantee that the system would not discover other proxies, and Amazon ultimately disbanded the team.

COMPAS (2016)

In May 2016, ProPublica reporters Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner published Machine Bias, an investigation of COMPAS, a recidivism prediction tool sold by Northpointe (now Equivant) and used in pretrial and sentencing decisions in several U.S. jurisdictions ^[8]. ProPublica's analysis of more than 7,000 risk scores from Broward County, Florida, found that Black defendants who did not go on to re-offend within two years were almost twice as likely as white defendants to be labeled high risk, while white defendants who did re-offend were more likely than Black defendants to be labeled low risk ^[8]. Northpointe responded that the tool was calibrated equally across race: at any given score, the probability of re-offense was similar for Black and white defendants. As Chouldechova (2017) ^[10] and Kleinberg, Mullainathan, and Raghavan (2017) ^[11] later proved formally, both sides were correct given their definitions, and the two definitions cannot be jointly satisfied when base rates differ across groups. The case became the standard example of how proxies and base-rate differences make statistical fairness genuinely hard rather than merely a matter of careful engineering.

Apple Card (2019)

In November 2019, software developer David Heinemeier Hansson tweeted that the Apple Card, issued by Goldman Sachs, had given him a credit limit twenty times higher than his wife's, despite shared assets and her higher credit score. The complaint went viral, drew similar accounts from other couples (including Steve Wozniak), and prompted an investigation by the New York Department of Financial Services ^[24]. The DFS report issued in March 2021 did not find a violation of fair lending laws, but the superintendent noted that the case showed how nontransparent algorithmic credit models can erode public trust and that consumer credit laws need updating ^[27]. The episode is often cited as an example of how a model can produce a gendered outcome through proxies even when sex is excluded from the inputs, and how difficult it is for regulators to investigate proprietary algorithms.

Healthcare risk prediction (Obermeyer et al., 2019)

Obermeyer, Powers, Vogeli, and Mullainathan published Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations in Science in October 2019 ^[23]. They studied a commercial population health tool used by U.S. health systems to identify patients for extra care management. The model used predicted future healthcare costs as a stand-in for medical need. Because Black patients with the same level of illness historically generated lower spending (due to access barriers, mistrust, and other factors), the algorithm assigned Black patients lower risk scores at every level of measured illness. At a given algorithmic risk score, Black patients had 26.3 percent more chronic conditions than white patients. The authors estimated that correcting the bias would more than double the share of Black patients automatically identified for extra care, from 17.7 to 46.5 percent. Cost was a proxy for need, and the proxy was systematically worse for one group. Working with the manufacturer, the researchers showed that switching to direct measures of health reduced the disparity by about 84 percent ^[23].

Online ad targeting (Datta, Tschantz, Datta, 2015)

Using an automated experimentation tool called AdFisher, Carnegie Mellon researchers Amit Datta, Michael Carl Tschantz, and Anupam Datta created 1,000 simulated browsing agents, half labeled male and half labeled female in Google's ad settings ^[6]. After visiting a fixed set of employment websites, the male-labeled agents were shown ads for high-paying executive coaching services much more often than the female-labeled agents ^[6]. The study did not establish whether the disparity came from advertiser targeting choices, Google's optimization, or upstream training data, but it showed that an opaque ad-targeting system was producing gendered outcomes that would be illegal in many employment contexts.

How do fairness theories treat proxies?

Disparate treatment versus disparate impact

U.S. anti-discrimination doctrine recognizes two routes to a finding of discrimination. Disparate treatment is intentional differential treatment of individuals because of a protected characteristic. Disparate impact, established in Griggs v. Duke Power Co. (1971), refers to a facially neutral practice that disproportionately harms a protected group and is not justified by business necessity. Proxy variables sit awkwardly between the two doctrines. A model that uses a proxy without intent might produce disparate impact without disparate treatment. A model whose designers chose a proxy specifically to track race or sex can constitute disparate treatment by proxy, a form of intentional discrimination disguised as a neutral feature. Courts and agencies have addressed the latter, for example in housing and lending guidance from the Department of Housing and Urban Development and the Consumer Financial Protection Bureau.

Why does removing protected attributes not remove bias?

The naive approach of removing the protected attribute from the training data, sometimes called fairness through unawareness, assumes that a model that never sees S cannot discriminate on S. Dwork and colleagues argued in Fairness Through Awareness (2012) that this is false in any realistic setting because proxies make S effectively recoverable from the remaining features ^[2]. Barocas, Hardt, and Narayanan put the mechanism plainly in their textbook Fairness and Machine Learning: "If a classifier trained on the original data uses the sensitive attribute and we remove the attribute, the classifier will then find a redundant encoding in terms of the other features." ^[30] In a large feature space the sensitive attribute is generally redundant given the other features, so blinding the model to it does not blind the model to the pattern. Hardt's later work and the broader literature reached the same conclusion: meaningful fairness in practice requires using S during evaluation, and often during training, rather than hiding it ^[9][30].

What statistical fairness criteria exist?

Researchers have proposed many group-level fairness criteria. The three most discussed are summarized below.

Scroll sideways for more →

Chouldechova (2017) ^[10] and Kleinberg, Mullainathan, and Raghavan (2017) ^[11] independently proved that calibration and equalized odds (or, equivalently in the binary case, balance in error rates) cannot both hold across groups when the base rates of the outcome differ. This is a genuine impossibility result, not a matter of better engineering. It means that any system that meets one fairness criterion in a setting with unequal base rates will violate another, and the choice between them is normative.

Causal fairness

Kusner, Loftus, Russell, and Silva introduced Counterfactual Fairness in 2017, defining a model as fair toward an individual if its prediction would have been the same in the counterfactual world where the individual's protected attribute had been different ^[12]. Kilbertus and colleagues' Avoiding Discrimination Through Causal Reasoning (2017) proposed a related causal-graph-based framework distinguishing legitimate from illegitimate paths between S and the prediction ^[13]. Both approaches require explicit assumptions about the data-generating process, which are hard to verify, but they offer a more principled treatment of proxies than purely observational criteria.

How do you detect proxy variables?

Because proxies can be subtle, practitioners use a small toolkit of statistical and adversarial methods to flag them.

Mutual information and conditional entropy quantify how much information a feature carries about S. Features with high mutual information are likely strong proxies. Pairwise correlations, regression coefficients, and chi-squared tests serve the same purpose for simpler cases.

Predictive auditing trains a separate model to predict S from the remaining features. If S can be predicted with high accuracy from the input set, then strong proxies are present. This procedure scales to large feature sets and works for unstructured inputs such as text and images.

Adversarial probing extends this idea inside the model. An auxiliary network attempts to recover S from intermediate representations. If it succeeds, proxies are encoded in the representation, and the model can in principle act on them.

Causal auditing inspects the assumed graph of relationships between features, S, and the outcome. Paths from S to the prediction that pass through legitimate explanatory variables (for example, S to education to job performance, where education is a legitimate qualification) may be acceptable, while direct paths or paths through non-explanatory proxies are not.

No single test is decisive. Strong proxies can hide in interactions, in nonlinear combinations of weak features, or in latent representations learned by deep models. Detection is an ongoing audit rather than a one-time check.

How can proxy bias be mitigated?

Fairness mitigation methods are usually grouped by where they intervene in the modeling pipeline.

Scroll sideways for more →

Pre-processing methods change the training data so that a downstream model trained on it produces fairer predictions, often by reweighting samples or learning a representation in which S cannot be predicted. In-processing methods modify the loss function or the optimization problem, for example by adding a regularizer that penalizes correlation between predictions and S, or by training the model jointly with an adversary that tries to recover S from its outputs. Post-processing methods take a fixed model and adjust its predictions or thresholds per group to satisfy a chosen fairness criterion.

Each approach has tradeoffs. Pre-processing is portable across downstream models but may degrade utility because the downstream model never sees the raw data. In-processing can deliver better fairness-utility tradeoffs but requires retraining and access to S during training. Post-processing is the simplest to deploy but requires using S at decision time, which may be legally restricted in domains such as U.S. consumer credit, where the Equal Credit Opportunity Act forbids using sex or race in the lending decision itself.

What tools detect and mitigate proxy bias?

Several open source libraries package these algorithms with diagnostics and dashboards.

AIF360 (AI Fairness 360), released by IBM Research in 2018 with an accompanying paper by Bellamy and colleagues, includes more than seventy fairness metrics and a dozen mitigation algorithms covering pre-, in-, and post-processing ^[17].

Fairlearn, originally developed at Microsoft and described in a 2020 white paper led by Sarah Bird, focuses on group fairness assessment and mitigation in classification and regression, with an interactive dashboard and integrations for scikit-learn pipelines ^[25]. The project moved to an independent open source governance model in 2021.

Google's What-If Tool and Fairness Indicators support interactive slicing of model performance by demographic subgroup and integrate with TensorFlow Model Analysis.

Aequitas, released in 2018 by the University of Chicago's Center for Data Science and Public Policy under Pedro Saleiro and colleagues, targets policymakers and auditors and emphasizes group bias and disparity reports ^[20].

What are the limits of proxy mitigation?

Proxy mitigation faces several genuine limits.

First, no algorithm can remove sensitive information from data in which proxies are very predictive. If a feature set lets a separate model predict S with 98 percent accuracy, then any downstream classifier with sufficient capacity can also recover S, and removing or reweighting features will only force the model to rely on subtler combinations of the same signal.

Second, fairness criteria conflict. Calibration, equalized odds, and demographic parity cannot in general all be satisfied simultaneously when group base rates differ. Choosing a criterion is a policy decision, not a technical one, and different stakeholders will reasonably prefer different choices.

Third, group fairness is not the same as individual fairness. A model can satisfy demographic parity at the group level while still producing arbitrary, idiosyncratic decisions for individuals.

Fourth, statistical fairness is not the same as causal fairness. A model that satisfies equalized odds may still discriminate through pathways that a causal analysis would judge illegitimate, and conversely a counterfactually fair model may fail group statistics.

Fifth, the legal and the statistical can pull in opposite directions. The Equal Credit Opportunity Act and similar laws restrict the use of sensitive attributes in decisions, while many of the most effective fairness algorithms require those attributes either at training time, at decision time, or both. Lawyers, regulators, and engineers continue to negotiate this tension, and the answers vary by jurisdiction and product.

Sixth, fairness can trade off with accuracy, though the size of the tradeoff depends on the dataset and the criterion. In some cases, a model can be made fairer without losing meaningful accuracy. In other cases, particularly when the training labels themselves reflect historical bias, a meaningful gain in group fairness requires accepting lower aggregate predictive accuracy on those biased labels.

How do proxies affect large language models?

Large language models complicate the proxy story in two ways. First, their training data contains rich correlational structure between linguistic features and demographics, so the models pick up associations between, for example, names, dialects, occupations, and stereotypes. Bender, Gebru, McMillan-Major, and Shmitchell argued in On the Dangers of Stochastic Parrots (2021) that the size and opacity of modern LLMs makes it hard to audit which associations they have absorbed, and that the costs of fixing biased outputs after deployment are high ^[26]. Second, LLMs are increasingly used as components in pipelines for screening resumes, generating clinical summaries, or making recommendations, where their proxy-laden outputs feed into downstream decisions that are subject to anti-discrimination law.

Generative models also raise new privacy attacks. Synthetic data generated to protect individual privacy can still leak attributes about the source population, including sensitive ones, through proxies in the generated samples. Membership inference and attribute inference attacks exploit precisely the kind of proxy structure that fair-ML researchers try to suppress.

The regulatory landscape has tightened. The EU AI Act requires high-risk AI providers to examine training data for bias and to test systems for discriminatory outcomes ^[28]. New York City's Local Law 144, in effect since 2023, requires employers using automated employment decision tools to commission an independent bias audit and disclose the results. The Colorado AI Act of 2024 imposes similar obligations on developers and deployers of high-risk systems. In the United States, federal agencies including the EEOC, CFPB, FTC, and DOJ issued a joint statement in April 2023 affirming that existing anti-discrimination laws apply to AI-driven systems ^[29].

What is the best practice for handling proxies?

A reasonable proxy-aware development checklist looks something like the following.

Document the dataset using a Datasheet for Datasets (Gebru et al., 2018). Datasheets describe the dataset's motivation, composition, collection process, recommended uses, and known limitations, and they make proxy structure easier to spot during downstream review ^[19].

Document the model using Model Cards (Mitchell et al., 2019) ^[22]. A model card reports performance disaggregated by demographic group, intended uses, and known failure modes.

Audit for proxies before training using mutual information, predictive auditing, or causal analysis. Disclose strong proxies in the model card.

Choose a fairness criterion in consultation with domain experts and legal counsel, recognizing that the choice is normative.

Evaluate fairness with the same rigor as accuracy, including subgroup metrics, intersectional slices, and confidence intervals.

Monitor in production. Distribution shift can introduce new proxies that did not exist at training time, and feedback loops (for example, in predictive policing) can amplify bias over time.

Consider bias bounty programs, in which external researchers are paid to find unexpected fairness failures, modeled on security bug bounties. Twitter ran an early image-cropping bias bounty in 2021, and similar programs have followed at OpenAI and elsewhere.

See also

• Algorithmic fairness
• Fairness
• Bias
• Discrimination
• Disparate impact
• Disparate treatment
• Redlining
• Coded bias
• Equalized odds
• Demographic parity
• AIF360
• ML credit scoring
• Recidivism prediction
• COMPAS

References

1. Bertrand, M., and Mullainathan, S. (2004). "Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination." *American Economic Review*, 94(4), 991 to 1013. ↩
2. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). "Fairness Through Awareness." *Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS '12)*, 214 to 226. ↩
3. Kamiran, F., and Calders, T. (2012). "Data preprocessing techniques for classification without discrimination." *Knowledge and Information Systems*, 33(1), 1 to 33.
4. Kamishima, T., Akaho, S., Asoh, H., and Sakuma, J. (2012). "Fairness-Aware Classifier with Prejudice Remover Regularizer." *ECML PKDD 2012*, Part II, LNCS 7524, 35 to 50.
5. Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013). "Learning Fair Representations." *Proceedings of the 30th International Conference on Machine Learning (ICML)*, PMLR 28(3), 325 to 333.
6. Datta, A., Tschantz, M. C., and Datta, A. (2015). "Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination." *Proceedings on Privacy Enhancing Technologies*, 2015(1), 92 to 112. ↩
7. Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015). "Certifying and removing disparate impact." *Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 259 to 268.
8. Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). "Machine Bias." ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing ↩
9. Hardt, M., Price, E., and Srebro, N. (2016). "Equality of Opportunity in Supervised Learning." *Advances in Neural Information Processing Systems 29 (NeurIPS)*, 3315 to 3323. ↩
10. Chouldechova, A. (2017). "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." *Big Data*, 5(2), 153 to 163. ↩
11. Kleinberg, J., Mullainathan, S., and Raghavan, M. (2017). "Inherent Trade-Offs in the Fair Determination of Risk Scores." *Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS)*, 43:1 to 43:23. ↩
12. Kusner, M. J., Loftus, J. R., Russell, C., and Silva, R. (2017). "Counterfactual Fairness." *Advances in Neural Information Processing Systems 30 (NeurIPS)*, 4066 to 4076. ↩
13. Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., and Schoelkopf, B. (2017). "Avoiding Discrimination Through Causal Reasoning." *Advances in Neural Information Processing Systems 30 (NeurIPS)*, 656 to 666. ↩
14. Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., and Weinberger, K. Q. (2017). "On Fairness and Calibration." *Advances in Neural Information Processing Systems 30 (NeurIPS)*, 5680 to 5689.
15. Rothstein, R. (2017). *The Color of Law: A Forgotten History of How Our Government Segregated America*. Liveright. ↩
16. Zafar, M. B., Valera, I., Rodriguez, M. G., and Gummadi, K. P. (2017). "Fairness Constraints: Mechanisms for Fair Classification." *Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS)*, PMLR 54, 962 to 970.
17. Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K. N., Richards, J., Saha, D., Sattigeri, P., Singh, M., Varshney, K. R., and Zhang, Y. (2018). "AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias." arXiv:1810.01943. ↩
18. Dastin, J. (2018). "Amazon scraps secret AI recruiting tool that showed bias against women." Reuters, October 10, 2018. ↩
19. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daume III, H., and Crawford, K. (2018). "Datasheets for Datasets." arXiv:1803.09010. Later published in *Communications of the ACM*, 64(12), 2021. ↩
20. Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., and Ghani, R. (2018). "Aequitas: A Bias and Fairness Audit Toolkit." arXiv:1811.05577. ↩
21. Zhang, B. H., Lemoine, B., and Mitchell, M. (2018). "Mitigating Unwanted Biases with Adversarial Learning." *Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society*, 335 to 340.
22. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019). "Model Cards for Model Reporting." *Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*)*, 220 to 229. ↩
23. Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). "Dissecting racial bias in an algorithm used to manage the health of populations." *Science*, 366(6464), 447 to 453. ↩
24. Vigdor, N. (2019). "Apple Card Investigated After Gender Discrimination Complaints." *The New York Times*, November 10, 2019. ↩
25. Bird, S., Dudik, M., Edgar, R., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., and Walker, K. (2020). "Fairlearn: A toolkit for assessing and improving fairness in AI." Microsoft Research Technical Report MSR-TR-2020-32. ↩
26. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" *Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT)*, 610 to 623. ↩
27. New York Department of Financial Services (2021). "Report on Apple Card Investigation." March 23, 2021. ↩
28. European Parliament and Council (2024). "Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act)." Official Journal of the European Union, July 12, 2024. ↩
29. Equal Employment Opportunity Commission, Consumer Financial Protection Bureau, Department of Justice, and Federal Trade Commission (2023). "Joint Statement on Enforcement Efforts Against Discrimination and Bias in Automated Systems." April 25, 2023. ↩
30. Barocas, S., Hardt, M., and Narayanan, A. (2023). *Fairness and Machine Learning: Limitations and Opportunities*. MIT Press. Available at fairmlbook.org. See the "No fairness through unawareness" discussion in the Classification chapter. ↩
31. Consumer Financial Protection Bureau (2014). "Using publicly available information to proxy for unidentified race and ethnicity: A methodology and assessment." CFPB Office of Research, Summer 2014. ↩