Civil rights groups concerned about biases in recidivism reduction algorithm

An algorithm that was launched in June by the U.S. Department of Justice to predict the likelihood of recidivism for federal inmates is being criticized by civil rights activists of possible gender and racial bias.

The Prisoner Assessment Tool Targeting Estimated Risk and Needs (PATTERN) algorithm is part of the First Step Act, a criminal justice reform bill that was passed by Congress with bipartisan support and signed into law by President Donald Trump in December 2018. The law intends to reduce recidivism and streamline the process through which inmates can be rewarded for good behavior, with the ultimate goal of reducing the federal prison population.

According to the DOJ, 3,100 federal inmates were already released as a result of the First Step Act. According to the Bureau of Prisons, as of September 12, there were 177,080 inmates under custody of the BOP, either in federal or private facilities.

The way PATTERN works is by classifying a BOP prisoner into one of four Risk Level Categories (“RLCs”) by scoring them by assigning points in 17 different categories. In their report explaining how the algorithm works, the DOJ touts that the new algorithm is 15% more accurate than similar tools, according to a metric called Area Under the Curve (AUC). The AUC represents the likelihood that the algorithm would give any given recidivist a higher risk score than a non-recidivist.

But as with other algorithms used by the justice system, PATTERN is raising controversy over the types of inputs it uses. Earlier this month, civil rights groups have published an open letter criticizing the fact that the algorithm uses historical data to calculate the assessments, which would make it a fertile ground for biases.

While the DOJ report also explains that tests were conducted to assess “racial and ethnic neutrality,” and that there is “minimal racial/ethnic disparity for PATTERN’s prediction strength,” the civil rights groups also urged the DOJ to address concerns about “racial, ethnic and gender biases.” According to the authors of the letter, failures in the algorithm “could be holding back thousands more from the freedom they deserve.”

An additional concern is that, whatever solutions are created to reduce the number of federal inmates, that would only impact a small proportion of prisoners in the country. The 2018 report on mass incarceration by the Prison Policy Initiative shows that only 10% of inmates in the United States are in federal facilities, compared to 60% for state prisons and 30% for local jails.

The Bureau of Justice Statistics reported this year that there were almost 2.2 million inmates in the United States in 2016, which means that for every 100,000 people residing in the United States, approximately 670 of them were behind bars. According to the Vera Institute of Justice, incarceration costs an average of more than $31,000 per inmate, per year, nationwide. In some states, it’s as much as $60,000. 

Nursing Home Compare: making sense of a crucial tool for the future

The Five-Star Quality Rating System is a convenient and widely used tool created by the Center for Medicare and Medicaid Services (CMS) to help people find good nursing homes, but it can be a subject of controversy when it comes to specific metrics that it uses. 

In 1998, the Center for Medicare and Medicaid Services (CMS) initially launched Nursing Home Compare as a federal website for information about nursing homes. The Five-Star Quality Rating system was an update made in 2008 to help the public better identify and compare nursing homes based on an automated score generated by specific metrics of the nursing homes. The Nursing Home Compare website works by requesting the zip code of the user and listing all available nursing homes within a 25 mile radius. The user can then select up to three to compare the ratings based on reports certified by the CMS (see screenshot below). Providers are given a score from one to five stars on criteria such as health inspections, staffing and quality measures, where one star represents below average and five star represents above average. 

Nursing Home Compare: making sense of a crucial tool for the future

While the star rating method is an easy way for consumers to visualize differences in quality between nursing homes, there might be issues which could threaten the validity of the assessment.

In an academic study published in 2018, researchers found that higher ratings on Nursing Home Compare did not directly translate to a lower rate of hospitalization, deeming the Five-Star Quality Rating System to be “less meaningful as an indicator of nursing home quality for post-acute care patients.” This study raises the concern that the rating system might not be incentivizing nursing homes to focus on patient care, but rather on simply meeting the standards that are checked during inspections in order to receive a higher rating. 

Another criticism regarding Nursing Home Compare is the lack of understanding towards assessing overall quality of nursing homes. In the most recent change to the system, a facility will automatically be assigned a one-star rating for the “registered nurse staffing” category if it reports four or more days per quarter with no registered nurse on site. The CMS said this change was to reflect that “nurse staffing has the greatest impact on the quality of care nursing homes deliver.” This change, while positive, was met with criticism from the American Health Care Association (AHCA). In a statement, the AHCA says that “the staff rating still does not include therapists” who also play a critical role in ensuring patient-centered care. While the CMS is continuously making strides to improve the system, they may well  be overlooking other vital components of quality care. 

The most recent study by the CDC in 2016 found that there were about 15,600 nursing homes in the U.S., however, that number is dropping. In an interview with the New York Times, University of Pittsburgh health policy researcher Dr. Nicholas Castle said that 200 to 300 nursing homes close each year due to declining number of residents who might choose alternatives such as assisted living or other ways to stay at home. Meanwhile, the Population Reference Bureau projects that by 2060, nearly 100 million Americans will be 65 or older, which is more than double the number from 2016.  Putting these pieces of information together, there will be an inevitable rise in demand for better nursing home facilities, and for an accurate rating system to help people find those homes. 

Could CMS’ Fraud Prevention System be unfair?

The Centers for Medicare and Medicaid Services (CMS) uses the Fraud Prevention System (FPS) to detect improper Medicare payments by processing millions of fee-for-service claims every day. But the focus on monetary return might cause the system to focus on some fraudulent healthcare providers in lieu of others.

FPS analyzes data related to Medicare fee-for-service claims to detect improper claims and develop leads for fraud investigations. The large number of irregular claims shows the need for such a system: CMS estimates that 9.51% of its payments in the 2017 fiscal year were towards claims that violated Medicare policy, which translates to $36.21 billion in improper payments. Past years also had high improper payment rates: in the 2016 fiscal year, it was 11%, and in the 2015 fiscal year, it was 12.1%.

These losses are the reason why the Small Business Jobs Act of 2010 required the CMS to create the FPS, which uses predictive analytics to process all fee-for-service claims prior to payment and prevent the payment of improper claims.

The issue is that FPS has a huge impact, both in the operation of Medicare and the lives of millions of Americans.  According to Northrop Grumman, the defense contractor chosen to implement the FPS, the system includes 42 million Medicare beneficiary profiles and 2 million healthcare provider profiles as of 2013, with millions of claims processed daily. Even though the system was expensive to implement, costing around $51.7 million in its fourth implementation year (2015), the investment is yielding results. In 2015, it helped identify or prevent $654.8 million of improper payment, which means approximately a $11.5 return for every dollar invested, although the rate of return was not as high in previous years.

Although it is important to consider the return on investment when evaluating the FPS system, using that as the only measurement of success can lead to undesirable biases. A report to Congress mentioned the potential side effects of concentrating on return on investment: the program might focus on getting money back from “amateur” fraudsters rather than “professional” fraudsters who, for example, might offshore their illegitimate gains. An “amateur” fraudster might be a healthcare provider that treats real patients but also makes improper claims. It would be easier to get money back from this business rather than someone who offshored all the money. In the short run, this could mean using fewer resources and getting higher returns, which would look good in terms of return on investment. However, the professional fraudsters are also a problem, and it’s important to go after them as well. Is it fair to target “small” fraud instead of “big” fraud just because the money is easier to recover?

Furthermore, healthcare providers can be punished over errors even when most of their fee-for-service claims are legitimate. For example, CMS revoked the billing privileges of the company Arriva (which describes itself as “the nation’s largest supplier of home-delivered diabetic testing supplies”) based on 211 improper claims, which represents only 0.003% of its claims over the past five years. Even though this reduces healthcare fraud, the decision also affects Arriva’s ability to provide real medical service to its customers.

To investigate this issue, researchers and journalists could view CMS’s Return on Investment reports and the Government Accountability Office’s reports on the Fraud Prevention System.

Investigation into this algorithm can start by contacting the press contacts of the Centers of Medicare and Medicaid Services, as well as people involved in health care fraud cases, since the FPS generates a lot of leads for investigations. The FBI also has a website dedicated to health care fraud news.


Algorithmic decision-making systems (ADMs) now influence many facets of our lives. Whether it be in finance, employment, welfare management, romance, or dynamic pricing, these systems are practically ubiquitous throughout the public and private sectors.

Operating at scale ADMs can impact large swaths of people—for better or worse. Algorithmic accountability reporting has emerged as a response, an attempt to uncover the power wielded by ADMs and detail their biases, mistakes, or misuse. Algorithmic accountability entails understanding how and when people exercise power within and through an algorithmic system, and on whose behalf.

Algorithm Tips hopes to stimulate algorithmic accountability reporting and support a robust reporting beat on algorithms, in particular by making it easier to investigate algorithms used in government decision making.

To do this we curate a database of ADMs in the US federal government which are of potential interest for investigation (and hope to expand to local and international jurisdictions in the future). On our home page you can search the database for interesting algorithms using keywords relating to facets such as agency (e.g., Dept. of Justice) or topic (e.g., health, police, etc.). Next, on our resources page, you can learn how to submit public records requests about algorithms, or find news articles and research papers about the uses and risks of algorithms. We hope you can get some inspiration there. And finally, we actually dig into some of these leads ourselves and post write-ups to our blog. We hope that journalists and other stakeholders can build on these posts and develop even deeper investigations.

Algorithm Tips is a project of the Northwestern University Computational Journalism Lab. If you have any questions, comments, or concerns (or want to talk about how to help us expand the effort), get in touch: Thanks!