COMPAS (recidivism risk assessment)

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a proprietary actuarial risk-assessment instrument used by United States courts and corrections agencies to estimate the likelihood that a criminal defendant will reoffend, fail to appear at a court date, or commit a violent crime. The tool is sold by Equivant, the present-day brand of the company originally known as Northpointe, Inc., which built the first version in the late 1990s ^[1][16]. COMPAS scores defendants on a one-to-ten scale across multiple risk dimensions, and these scores are used in pretrial release decisions, probation supervision, parole eligibility, and, in some jurisdictions, in sentencing. It is best known as the subject of a May 2016 ProPublica investigation that reported the tool produced higher false-positive rates for Black defendants and higher false-negative rates for white defendants, a finding the vendor disputed, and the resulting debate became a foundational case study in algorithmic fairness ^[1][2].

What sets COMPAS apart from the dozens of other risk instruments in use across the country is what happened to it in 2016. On May 23 of that year, ProPublica published Machine Bias, an investigation by Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner that analyzed more than 7,000 COMPAS scores from Broward County, Florida, and concluded that the algorithm made systematic mistakes by race ^[1]. According to ProPublica, Black defendants who did not go on to reoffend were nearly twice as likely to be tagged high-risk as comparable white defendants, and white defendants who did go on to reoffend were nearly twice as likely to be tagged low-risk ^[1]. Northpointe pushed back, arguing that the system was calibrated correctly within each racial group and had similar predictive accuracy across races ^[3]. Subsequent analysis showed that both characterizations could be true at the same time because they measure different things, which is the heart of why COMPAS became the founding case study of an entire research field ^[4][5].

In the years that followed, the COMPAS story drove the emergence of algorithmic fairness as a recognizable academic discipline, prompted formal impossibility theorems showing why the ProPublica and Northpointe positions could not be reconciled ^[4][5], anchored a wave of popular books on algorithmic bias ^[11], survived a Wisconsin Supreme Court challenge in Loomis v. Wisconsin ^[10], and helped seed the founding of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) in 2018. Almost every introduction to fairness in machine learning mentions COMPAS within the first few pages.

Who makes COMPAS? Northpointe and Equivant

Northpointe, Inc. was founded in 1989 by criminologists Tim Brennan and Dave Wells, with William Dieterich joining as a senior researcher ^[16]. The company built decision-support software for correctional agencies, drawing on Brennan's academic work on offender classification at the University of Colorado. The first version of the COMPAS instrument appeared in 1998 ^[16], and the suite expanded over the next decade to include separate scales for community supervision, reentry, women, and youth.

Northpointe was later acquired by Constellation Software, a Toronto-based holding company that owns dozens of vertical-market software firms. The acquisition placed Northpointe inside a portfolio that already included CourtView Justice Solutions and Constellation Justice Systems. In January 2017, the three sister companies were merged under a single brand called Equivant ^[16]. The rebrand was done partly for marketing reasons (so that the same sales team did not visit a county under three different names) and partly to put some distance between the corporate identity and the Northpointe name, which by that point had become inseparable from the ProPublica controversy.

Equivant continues to license the COMPAS suite to state corrections departments, county jails, probation offices, and pretrial services agencies. The company does not publish its customer list, and academic researchers who want to validate the instrument typically have to obtain scores through public-records requests directed at the agencies that use it, which is the route ProPublica took ^[2].

What is in the COMPAS instrument?

The core COMPAS questionnaire used in most deployments contains roughly 137 items ^[7]. The items combine information drawn from the criminal record (prior arrests, prior convictions, prior incarceration, age at first offense) with answers given by the defendant during an intake interview (employment history, family background, neighborhood characteristics, attitudes toward authority, peer associations, perceived opportunities for crime).

Race is not directly asked. This is the point Equivant returns to whenever the instrument is described as racially biased. Critics counter that several items act as proxies for race in the United States given how segregated neighborhoods, schooling, and policing patterns are, so omitting the variable does not insulate the score.

The questionnaire feeds into a set of separate scales. The three most cited in the academic literature are:

Scroll sideways for more →

Each scale produces a decile score from 1 to 10, with 1 to 4 conventionally treated as low risk, 5 to 7 as medium risk, and 8 to 10 as high risk. The thresholds are not strict cutoffs in the underlying mathematics; they are presentation conventions used in the printed reports that judges and probation officers see. The same questionnaire also produces a longer set of "criminogenic needs" scales (substance abuse, family criminality, antisocial peers, and others) that are intended for case planning rather than for risk gating.

How is COMPAS used in United States criminal justice?

COMPAS is one of several major risk-assessment products used in American criminal justice. Others include the Public Safety Assessment (PSA) developed by the Laura and John Arnold Foundation, the Level of Service / Case Management Inventory (LS/CMI) made by Multi-Health Systems, the Static-99 used for sex-offender risk, and an assortment of state-built tools. Adoption of COMPAS specifically is uneven and changes year to year, but in the early 2010s it was used at some level in Wisconsin, New York, California, Florida, Michigan, and several other states. After 2016 some jurisdictions replaced it with the PSA or with home-grown alternatives, and a few moved away from algorithmic risk tools altogether at the pretrial stage.

The most consequential use is at sentencing, where the score appears in the presentence investigation report (PSI) that the judge reads before imposing a term. This is also where the legal challenges have concentrated, because in pretrial decisions the judge is making a fast bail call and in probation the score affects supervision intensity rather than incarceration directly, but at sentencing a high score can plausibly contribute to a longer prison term. Wisconsin, where the Loomis case originated, used the instrument in this way ^[10].

What did ProPublica find? The 2016 investigation

The Broward County data was the right test bed for several reasons. Florida has a robust public-records statute, the county sheriff's office had been using COMPAS for several years and was willing to release scores under the statute, and the county is large and demographically mixed, so there was enough data on both Black and white defendants to do a meaningful subgroup analysis. ProPublica obtained more than 18,000 scores covering people booked into the Broward County jail between 2013 and 2014, and after merging with state and federal criminal-history records and excluding cases that lacked the required follow-up, the analysis settled on a usable cohort of 7,214 defendants ^[1][2].

Recidivism was defined operationally as a new arrest for a misdemeanor or felony within two years of the date of the COMPAS assessment ^[2]. The reporters then compared the predicted risk score to whether the person was actually re-arrested in that window.

ProPublica summarized the disparity directly: "The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants. White defendants were mislabeled as low risk more often than black defendants." ^[1]

The headline numbers from the ProPublica analysis are summarized in the table below. The two rows that drove the controversy are the false positives (high-risk score but no re-arrest) and false negatives (low-risk score but a subsequent re-arrest) ^[1].

Scroll sideways for more →

The pattern is striking when you set it next to the overall accuracy numbers. By ProPublica's account the model was about equally accurate at distinguishing future re-arrests from non-re-arrests in both groups, which is what the AUC numbers say. The errors it made, though, were sharply different. When the model was wrong about a Black defendant, it tended to be wrong in the direction of overestimating risk. When it was wrong about a white defendant, it tended to be wrong in the direction of underestimating risk. This is what ProPublica meant by machine bias ^[1].

ProPublica also published a methodology piece, How We Analyzed the COMPAS Recidivism Algorithm, which documented every step of the data cleaning, the regression specifications, and the cohort definitions ^[2]. This was unusual at the time for an investigative news story, and it made the analysis reproducible. Researchers have since rerun versions of it on the same dataset, often with small variations, and the qualitative findings have largely held up ^[9].

How did Northpointe respond? Predictive parity

Within weeks of the ProPublica story, Northpointe released a 39-page rebuttal authored by William Dieterich, Christina Mendoza, and Tim Brennan titled COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity ^[3]. The argument was technical, and a lot of the public confusion that followed comes from the fact that the technical argument was correct on its own terms.

Northpointe's claim was that COMPAS satisfied a fairness criterion called predictive parity, sometimes also called calibration. Predictive parity means that within any given risk score, the actual rate of re-arrest is approximately the same across racial groups. A defendant who scores a 7 has roughly the same chance of reoffending whether they are Black or white. From the perspective of someone using the score to make a decision, the score means the same thing in both groups. By this measure, the company argued, COMPAS was not biased ^[3].

ProPublica was measuring something else: balance for the false positive and false negative rates across groups. By the analysis that followed, the company and the reporters were both right; they were just measuring different things ^[4][5]. The deeper question, then, was whether the two definitions could be reconciled. The answer, it turned out, was no.

Why can the two fairness criteria not be reconciled? The impossibility theorems

In the months after the ProPublica piece, two papers appeared that formalized the trade-off and proved it was unavoidable.

Alexandra Chouldechova, then at Carnegie Mellon, published Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments in the journal Big Data in 2017 (a preprint had circulated earlier) ^[4]. The paper showed that whenever the base rate of recidivism differs between two groups (which it does in the COMPAS data, where the observed re-arrest rate among Black defendants in Broward was higher than among white defendants), an instrument that satisfies predictive parity cannot also have equal false positive and false negative rates across the groups ^[4]. The math is fairly compact once you write down the confusion matrix and the constraints, but the implication was sweeping: the ProPublica criterion and the Northpointe criterion are mutually incompatible, not because anyone designed the system poorly but because of an arithmetic identity.

Almost simultaneously, Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan posted Inherent Trade-Offs in the Fair Determination of Risk Scores (arXiv:1609.05807, later in ITCS 2017) ^[5]. Their result was more general. They identified three desirable fairness properties (calibration within groups, balance for the positive class, and balance for the negative class) and proved that "except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously" ^[5], with the constrained cases being those where the base rates are identical or where the predictor is perfect.

Taken together, the Chouldechova and Kleinberg results reframed the COMPAS debate. The question stopped being "is the algorithm biased" and started being "which kind of bias do we tolerate, given that we cannot avoid all of them at once." That is now the standard frame in algorithmic fairness, and many subsequent papers extend, sharpen, or relax the impossibility result under different assumptions ^[9].

Fairness criteria, side by side

Scroll sideways for more →

When base rates differ across groups, satisfying any one of these often forces violations of the others. This is the core lesson the field absorbed from the COMPAS dispute ^[4][5].

This tension also explains why simply hiding race from the model does not solve the problem. Approaches built on unawareness to a sensitive attribute, in which the protected variable is dropped from the inputs, do not by themselves guarantee equal error rates, because correlated features can carry the same information and the base-rate arithmetic still applies.

Is COMPAS even accurate? Dressel and Farid 2018

A separate strand of criticism asked a different question: forget bias for a moment, is COMPAS even good at predicting recidivism in the first place? In January 2018 Julia Dressel, then an undergraduate at Dartmouth, and Hany Farid, her advisor, published The Accuracy, Fairness, and Limits of Predicting Recidivism in Science Advances ^[7]. They ran two experiments on the ProPublica Broward data.

In the first, they showed short descriptions of defendants (sex, age, prior criminal history, current charge) to roughly 400 workers on Amazon Mechanical Turk and asked them to predict whether each person would be re-arrested within two years. The Turk workers, who had no criminal-justice expertise and were paid a few dollars, achieved a pooled accuracy of about 67 percent. COMPAS, drawing on its 137 features, achieved about 65 percent on the same defendants. The difference was not statistically significant ^[7].

In the second, the authors fit a simple logistic regression using only two features (the defendant's age and total number of prior convictions). It matched COMPAS performance ^[7].

The paper did not claim that human judgment is good at recidivism prediction. It claimed that COMPAS was not adding much over either humans or trivial models. That undercut a central justification for using a proprietary 137-item questionnaire at all. Equivant disputed the methodology, and a group of correctional researchers led by Anthony Flores published a rejoinder. The Dressel and Farid result, however, has been examined in several follow-up studies, and the broader conclusion (that the predictive ceiling on individual-level recidivism is fairly low and is reached by quite simple models) is now widely discussed in the field ^[7].

What did Loomis v. Wisconsin decide?

Eric Loomis was charged in 2013 with offenses related to a drive-by shooting in La Crosse, Wisconsin. He pleaded guilty to two of the lesser counts: attempting to flee a traffic officer and operating a motor vehicle without the owner's consent ^[10]. The presentence investigation report prepared by the Department of Corrections included a COMPAS assessment that scored him high on risk of pretrial recidivism, general recidivism, and violent recidivism. The judge cited the COMPAS report at sentencing, and Loomis was sentenced to six years in prison ^[10].

Loomis appealed, arguing that his due-process rights were violated because (1) the proprietary nature of COMPAS prevented him from examining how the score was produced, (2) the score was based in part on group statistics rather than individualized inquiry, and (3) the instrument used gender as an input ^[10].

The Wisconsin Supreme Court ruled in State v. Loomis, 881 N.W.2d 749 (Wis. 2016) that the use of COMPAS in sentencing did not violate due process, but only under specific conditions ^[10]. The court wrote that "if used properly, observing the limitations and cautions set forth herein, a circuit court's consideration of a COMPAS risk assessment at sentencing does not violate a defendant's right to due process" ^[10]. Among the conditions the court set:

• COMPAS scores cannot be the determinative factor in deciding whether to incarcerate or in setting the length of a sentence ^[10].
• A COMPAS score in a presentence investigation report must be accompanied by a written advisement that lists the tool's limitations, including its proprietary nature, its reliance on group data, and the fact that some studies have raised questions about how it classifies minority offenders ^[10].
• The use of gender as an input was permitted because, in the court's reasoning, it can improve accuracy ^[10].

The United States Supreme Court denied certiorari in 2017, leaving the Wisconsin ruling in place ^[10]. Loomis is now a frequently cited American legal authority on the use of proprietary risk instruments in sentencing. It permits the practice with caveats, but the caveats are unusually elaborate for a sentencing-procedure case, and the opinion has been heavily discussed in subsequent academic commentary ^[10].

How did COMPAS shape the algorithmic fairness field?

The COMPAS controversy did not invent the academic study of fairness in machine learning. There was earlier work by Cynthia Dwork on fairness through awareness, by Toon Calders and Sicco Verwer on classification under fairness constraints, and a small FAT/ML workshop series running since 2014. What COMPAS did was give the field a single, vivid, real-world case that everyone could refer to. Many fairness papers from 2017 onward include a benchmark experiment on the ProPublica Broward dataset, and it became one of the standard testbeds in libraries like AIF360.

Four effects on the field stand out:

First, the impossibility theorems by Chouldechova and Kleinberg et al. became foundational ^[4][5]. They are now taught early in most algorithmic fairness courses. The framing they introduced (multiple incompatible fairness criteria, choose your trade-off explicitly) has become a default frame.

Second, Moritz Hardt, Eric Price, and Nati Srebro's 2016 paper Equality of Opportunity in Supervised Learning introduced the equalized-odds criterion, which is intended to capture the kind of disparity ProPublica documented while remaining mathematically tractable ^[6]. It is now one of the most-cited fairness criteria.

Third, the conversation went mainstream. Cathy O'Neil's Weapons of Math Destruction came out in September 2016, four months after the ProPublica piece, with COMPAS as one of its central case studies ^[11]. Virginia Eubanks's Automating Inequality (2018) ^[12] and Safiya Noble's Algorithms of Oppression (2018) ^[13] extended the broader argument across welfare administration, child-protective services, and search engines. Julia Angwin went on to co-found The Markup, an investigative outlet focused on algorithmic accountability.

Fourth, the inaugural ACM Conference on Fairness, Accountability, and Transparency (then FAT*, now FAccT) was held in 2018. The conference grew out of the FAT/ML workshop series and was explicitly framed around the kind of problem that COMPAS exemplified. It has since become a central venue for fairness research.

What happened after 2018? Replications, reforms, and replacements

The years after 2018 produced more academic work, more legal and policy responses, and slow shifts in adoption.

Replication studies have largely confirmed the core ProPublica findings on the Broward data, while complicating some of the secondary claims. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq published Algorithmic Decision Making and the Cost of Fairness in 2017, formally analyzing the trade-off and arguing that some of ProPublica's framing conflated separate concepts ^[9]. Studies on data from other jurisdictions found similar patterns of differential error rates by race, though the magnitudes vary.

On the policy side, several jurisdictions stepped back from algorithmic risk tools at the pretrial stage. New Jersey shifted toward the Public Safety Assessment as part of a broader bail-reform package. Some California counties have used the PSA instead of COMPAS. New York State's 2019 bail reform reduced the use of pretrial detention generally, which made the choice of instrument less consequential at the front end. Other jurisdictions, including parts of Wisconsin, continue to use COMPAS.

Equivant has continued to refine and market the COMPAS suite, and the company has commissioned validation studies on its updated scales. Critics, including the Partnership on AI in its 2019 Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System, have argued that the validation studies are not enough on their own and that the deeper questions about base rates, ground-truth labels (re-arrest is not the same as re-offense), and the social meaning of risk scores remain unresolved ^[14].

The Partnership on AI report also pointed out something that gets lost in the technical debates. The labeled outcome in every COMPAS validation, ProPublica analysis, and academic replication is re-arrest, not actual recidivism ^[14]. Whether someone gets arrested again depends on policing patterns, which are themselves not race-neutral in many jurisdictions. So even a perfectly calibrated model trained on re-arrest data inherits the disparities in policing. This problem ("label bias") has become one of the active research directions in the field that COMPAS helped create.

Why does COMPAS still matter?

It would be a mistake to read the COMPAS story as a story about one bad algorithm. By most technical measures, COMPAS is unremarkable. It is a scoring tool with a moderately rich feature set and predictive performance comparable to other instruments and, by the Dressel and Farid result, to untrained humans ^[7]. The reason it became the case study it did is that ProPublica chose well, the data was available, the company defended the product publicly, and the resulting argument exposed a structural feature of risk prediction that nobody had fully articulated before: when you predict an outcome that has different base rates across groups, you are forced to choose which kind of fairness you want, because you cannot have all of them at once ^[4][5].

That lesson is now embedded in the way machine-learning practitioners think about deploying models in any high-stakes setting, including hiring, lending, healthcare, and child-welfare screening. Most of those deployments do not get a ProPublica-style audit. COMPAS got one, and the field is still working through what the audit revealed.

ELI5: COMPAS in simple terms

Imagine a computer program that reads a few facts about a person who has been arrested (their age, their past record, answers to some questions) and then gives them a number from 1 to 10 that is supposed to say how likely they are to get in trouble with the law again. Judges sometimes look at that number when they decide things like bail or how long a sentence should be.

In 2016, a news organization called ProPublica checked the program against what actually happened to thousands of people in Florida. They found that when the program guessed wrong, it tended to label Black people who did not reoffend as "high risk" more often, and it tended to label white people who did reoffend as "low risk" more often. The company that makes the program said it was fair in a different way: a 7 meant about the same chance of reoffending no matter the person's race.

It turned out both could be true, and mathematicians proved that you cannot make all the different kinds of "fair" happen at the same time when the groups have different overall rates of reoffending. That surprising fact is the main reason people still talk about COMPAS today.

See also

• Algorithmic fairness
• Algorithmic bias
• AI ethics
• AI safety
• AIF360
• Bias
• Calibration
• Equalized odds
• Unawareness to a sensitive attribute
• Logistic regression
• AUC
• FAccT
• Loomis v. Wisconsin
• ProPublica
• Northpointe
• Equivant
• Cathy O'Neil
• Weapons of Math Destruction
• Virginia Eubanks
• Safiya Noble
• Chouldechova
• Kleinberg, Mullainathan, Raghavan
• Recidivism
• Criminal justice
• Risk assessment
• Julia Angwin

References

1. Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). *Machine Bias: There's software used across the country to predict future criminals. And it's biased against blacks.* ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing ↩
2. Larson, J., Mattu, S., Kirchner, L., and Angwin, J. (2016). *How We Analyzed the COMPAS Recidivism Algorithm.* ProPublica, May 23, 2016. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm ↩
3. Dieterich, W., Mendoza, C., and Brennan, T. (2016). *COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity.* Northpointe Inc., July 2016. ↩
4. Chouldechova, A. (2017). *Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.* Big Data, 5(2), 153-163. https://doi.org/10.1089/big.2016.0047 ↩
5. Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). *Inherent Trade-Offs in the Fair Determination of Risk Scores.* arXiv:1609.05807. Published in ITCS 2017. https://arxiv.org/abs/1609.05807 ↩
6. Hardt, M., Price, E., and Srebro, N. (2016). *Equality of Opportunity in Supervised Learning.* NeurIPS 2016. ↩
7. Dressel, J. and Farid, H. (2018). *The Accuracy, Fairness, and Limits of Predicting Recidivism.* Science Advances, 4(1), eaao5580. https://doi.org/10.1126/sciadv.aao5580 ↩
8. Berk, R., Heidari, H., Jabbari, S., Kearns, M., and Roth, A. (2018). *Fairness in Criminal Justice Risk Assessments: The State of the Art.* Sociological Methods and Research, 50(1).
9. Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., and Huq, A. (2017). *Algorithmic Decision Making and the Cost of Fairness.* KDD 2017. ↩
10. *State v. Loomis*, 881 N.W.2d 749 (Wis. 2016), cert. denied (U.S. 2017). https://law.justia.com/cases/wisconsin/supreme-court/2016/2015ap000157-cr.html ↩
11. O'Neil, C. (2016). *Weapons of Math Destruction.* Crown Publishing. ↩
12. Eubanks, V. (2018). *Automating Inequality.* St. Martin's Press. ↩
13. Noble, S. U. (2018). *Algorithms of Oppression.* NYU Press. ↩
14. Partnership on AI (2019). *Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System.* ↩
15. Wikipedia contributors. *COMPAS (software).* Wikipedia. https://en.wikipedia.org/wiki/COMPAS_(software)
16. Equivant Corrections. *The History of The Northpointe Suite and How It Became Equivant Corrections.* ↩