Man and Machine: How Far Are We from AI Judges?

> Paper: Man and machine: AI and judicial decision making > Authors: Arthur Dyevre, Ahmad Shahvaroughi > arXiv: https://arxiv.org/abs/2603.19042

Opening: The First Case of Algorithmic Sentencing

In 2016, in Wisconsin, USA, a man named Eric Loomis stood in court facing a sentencing hearing after being arrested for driving a stolen vehicle.

Before ruling, the judge consulted a report from COMPAS, an AI system that scored Loomis as "high risk" with a high predicted likelihood of recidivism. The judge ultimately sentenced him to six years in prison. On appeal, Loomis's lawyers raised a striking question: is it unconstitutional to use an opaque algorithm to decide a person's freedom?

The Wisconsin Supreme Court ruled that using AI risk assessment tools is not inherently unconstitutional, but with conditions — the algorithm must not be the determining factor in sentencing, and defendants must be informed when it is used.

The Loomis case became a milestone in AI-driven justice, but it left a bigger question: can AI actually help judicial decision-making? How much? And at what cost? This is what Dyevre and Shahvaroughi set out to answer.

Background: The Rise of Judicial AI

From Experience-Based Judgment to Algorithmic Prediction

Judicial decisions — especially pretrial release and sentencing — have long been controversial. Traditional decision-making relies heavily on judicial discretion: the same crime may receive dramatically different sentences from different judges, fueling long-standing concerns about consistency and fairness.

Starting in the early 2000s, some US jurisdictions introduced risk assessment tools that use statistical models on defendants' records and demographics to predict future crime or failure to appear. Early tools were simple scorecards; with machine learning, they became more complex — and more of a "black box."

The COMPAS Controversy

In 2016, ProPublica published an investigation revealing racial bias in COMPAS:

• Black defendants were incorrectly flagged as "high risk" at twice the rate of white defendants
• White defendants were more often incorrectly flagged as "low risk," meaning genuinely dangerous white defendants could receive lighter sentences

Northpointe, COMPAS's developer, countered that ProPublica used the wrong fairness standard. Either way, the episode exposed a core dilemma: when an algorithm is trained on historically biased data, how do you prevent it from reproducing that bias?

A Systematic Review: What AI Actually Achieves in Court

The paper is a systematic review of empirical studies on AI in judicial decision-making.

Pretrial Decisions

Pretrial decisions cover bail approval (release vs. detention) and bail amounts.

Findings:

• AI tools can help judges better predict flight risk and recidivism risk
• However, when randomized controlled trials (RCTs) are introduced, the benefits often become weak or insignificant
• Some studies found no significant improvement in decision quality — and sometimes "algorithm aversion," where judges ignore AI advice entirely

Sentencing Decisions

Findings:

• AI's impact on sentencing is even more limited
• Many studies show negligible changes in sentencing outcomes after adopting risk assessments
• Sentencing involves too many normative judgments — not just risk prediction, but punishment, rehabilitation, deterrence, and justice for victims

Algorithmic Fairness

The paper discusses multiple dimensions of fairness:

1. Individual fairness: similar individuals should be treated similarly — but how to define "similar"? 2. Group fairness: different racial/gender groups should be treated equally — but statistical definitions of equality are numerous and often mutually incompatible. ProPublica emphasized false positive rate parity, while Northpointe emphasized predictive calibration; mathematically, both cannot hold simultaneously. 3. Procedural fairness: is the process transparent? Can defendants contest it? Deep learning models are often black boxes that cannot explain individual predictions.

Why AI Underperforms Expectations in Justice

1. Complexity of Judicial Decision-Making

Judicial decisions are not just prediction problems but normative problems. Predicting "70% recidivism probability" does not dictate a sentence length; severity, rehabilitation potential, deterrence, and victims' justice all require value judgments.

2. The Black-Box Problem

Modern AI models are often unexplainable. This creates:

• Trust issues: judges and defendants struggle to trust unexplainable decisions
• Error correction issues: when the algorithm errs, it is hard to find and fix the mistake

3. Stubborn Data Bias

AI models learn from historical data. If historical bias exists (research suggests it does), AI learns and may amplify it. Even removing "race" as an input leaves proxy variables — zip code, education, employment — highly correlated with race.

4. The Human-Machine Collaboration Dilemma

In reality:

• Over-reliance: some judges accept AI advice uncritically, outsourcing responsibility
• Complete disregard: others ignore it entirely due to distrust
• Selective use: some use it only when it confirms their intuition, reinforcing existing bias

Genuine human-machine collaboration requires careful interface design and training — both still lacking.

Implications and Future Directions

For Judicial Practice

1. Transparency first: prefer interpretable models (decision trees, linear models) over black-box deep learning 2. Fairness audits: regularly audit tools for racial, gender, and socioeconomic bias 3. Human-machine collaboration design: build better decision-support interfaces 4. Clear legal frameworks: AI can only assist, never replace, human judgment

For AI Research

The paper calls for interdisciplinary research: legal scholars understanding AI's limits, AI researchers understanding judicial complexity, sociologists studying long-term impacts, and philosophers examining "algorithmic justice."

Open Questions

• Can algorithms ever be truly "fair," or is fairness inherently a political problem technology cannot solve?
• If AI is more accurate at predicting recidivism, how much should it influence decisions?
• Who is responsible when AI errs — the developer, the judge, or the justice system?

Conclusion: Between Efficiency and Justice

The paper's conclusions are cautious and balanced:

• AI tools have potential, but it must not be exaggerated
• Algorithmic bias is a real challenge requiring serious attention
• Real progress requires interdisciplinary cooperation, not techno-optimism or techno-fear

We are unlikely to see an "AI judge" on the bench anytime soon. But AI tools — risk assessment, case retrieval, sentencing reference systems — will increasingly enter the courts. The key question: how do we design them to enhance rather than undermine judicial fairness?

As the paper emphasizes, justice must not only be done but be seen and understood to be done — whether achieved by humans alone or with machine assistance.

Further reading:

• Paper: https://arxiv.org/abs/2603.19042
• ProPublica investigation: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
• Related case: State v. Loomis (2016)
• Related concepts: algorithmic fairness, risk assessment, COMPAS, pretrial release, sentencing guidelines