The Statistics of Discrimination
Ph.D. thesis, Harvard University, 2026.
ProQuest: 3350281160.
Full text: PDF.
Abstract
Policymakers, regulators, courts, and the public increasingly rely on statistical evidence to understand and address human and machine bias in domains like lending, policing, hiring, and medicine. Empirical studies measuring discrimination, however, face both definitional and data challenges. First, mathematical definitions of bias often imperfectly capture legal and intuitive notions of discrimination, and seemingly natural measures can have counterintuitive implications that would harm the groups they are intended to protect. Second, discrimination is often most concerning in settings where data are limited: Key information about decisions, such as the factors on which they were based, is often unavailable or difficult to measure. This dissertation develops theory, methods, and applications across a variety of domains aimed at addressing these twin challenges in the measurement of discrimination.
Part I studies algorithmic discrimination. Machine decisions are simpler to analyze than human decisions in important respects—decision inputs are structured and usually observable, and the same inputs reliably yield the same decisions—but the meaning of algorithmic “discrimination” is often unclear. Chapter 2 critiques popular mathematical definitions of algorithmic fairness that seek to equalize model performance metrics like false positive rates across race, gender, or other characteristics. Because these measures suffer from inframarginality, a subtle statistical limitation, equalizing them typically requires making suboptimal decisions, including for protected groups. Chapters 3, 4, and 5 develop alternative approaches to measuring discrimination in algorithmic decisions. In contrast to the approaches critiqued in Chapter 2, Chapter 3 advocates for a consequentialist approach to fairness, proposing an algorithm that efficiently reduces inequality while balancing competing policy goals in lending. Chapters 4 and 5 draw on legal frameworks governing discrimination. Chapter 4 develops a risk-based measure of disparate impact, along with a regression-based framework for estimating it in observational settings. Motivated by emerging AI regulations, Chapter 5 adapts correspondence experiments, a behavioral science method for detecting discrimination, to audit large language models used in hiring.
Part II extends the methods of Part I to human decision-making. Unlike algorithms, human decisions are often poorly documented, difficult to reconstruct, and costly to manipulate, hindering the measurement of discrimination. We focus on applications to criminal justice, where these problems are particularly acute. Chapter 6 develops the robust outcome test, a method for measuring disparate impact in settings where little is known about the factors driving decisions, rendering the regression-based approach of Chapter 4 infeasible. By combining “benchmark” and “outcome” tests, standard but individually flawed methods of detecting discrimination, the robust outcome test yields strong statistical guarantees under a nonparametric assumption that empirical evidence suggests often holds in practice. Paralleling the correspondence experiments of Chapter 5, Chapter 7 develops a causal framework for estimating discrimination from observational data in multi-stage decision processes like prosecutorial charging decisions occurring downstream of potentially discriminatory earlier decisions like racially biased arrests. Finally, an alternative response to data limitations is leveraging novel data sources. Chapter 8 uses large-scale telematics data, second-by-second records of driving behavior for hundreds of thousands of individuals, to benchmark racial disparities in traffic enforcement against actual differences in speeding, disentangling discriminatory enforcement from differences in underlying behavior.