# Mitigating Omitted- and Included-Variable Bias in Estimates of Disparate Impact

Submitted, 2024.

ArXiv: 1809.05651.

## Abstract

Managers, employers, policymakers, and others often seek to understand whether
decisions are biased against certain groups. One popular analytic strategy is to
estimate disparities after adjusting for observed covariates, typically with a
regression model. This approach, however, suffers from two key statistical
challenges. First, omitted-variable bias can skew results if the model does not
adjust for all relevant factors; second, and conversely, included-variable
bias—a lesser-known phenomenon—can skew results if the set of covariates
includes irrelevant factors. Here we introduce a new, three-step statistical
method, which we call risk-adjusted regression, to address both concerns in
settings where decision makers have clearly measurable objectives. In the first
step, we use all available covariates to estimate the value, or inversely, the
*risk*, of taking a certain action, such as approving a loan application or
hiring a job candidate. Second, we measure disparities in decisions after
adjusting for these risk estimates alone, mitigating the problem of
included-variable bias. Finally, in the third step, we assess the sensitivity
of results to potential mismeasurement of risk, addressing concerns about
omitted-variable bias. To do so, we develop a novel, non-parametric sensitivity
analysis that yields tight bounds on the true disparity in terms of the average
gap between true and estimated risk—a single interpretable parameter that
facilitates credible estimates. We demonstrate this approach on a detailed
dataset of 2.2 million police stops of pedestrians in New York City, and show
that traditional statistical tests of discrimination can substantially
underestimate the magnitude of disparities.