Statistical Power and Search Intensity Bias in Hit Rates Tests of Discrimination

Abstract

Objectives

This study derives the statistical power of the common hit rates test of taste-based discrimination in police searches. It also identifies search intensity as a source of bias in the test and proposes a simple empirical adjustment to account for the bias.

Methods

Data simulations, along with two empirical applications to motor vehicle search data, display the practical importance of both statistical power and search intensity bias in the hit rates test.

Results

Statistical power varies markedly with the parameters of the application. Differential search intensity across groups will bias results, but the simple empirical adjustment provides a valid test when the data contain a discrete indicator of search intensity. In the empirical applications, unadjusted and adjusted tests differ in their conclusion of whether police discrimination exists.

Conclusions

For the presentation of multiple hit rates tests, statistical power should be reported with p-values. Theoretical bounds on search intensity bias are wide, and the bias can persist for any sample size. Hit rates tests should therefore be interpreted with caution when data contain no indicator of search intensity.

Appendix

A.1 Proofs

Proof of Result 1

From (5),

$$\begin{aligned} \frac{\displaystyle \partial \Omega (\delta ,\gamma ,p)}{\displaystyle \partial \delta }=\frac{2 \delta p (1-p) (\gamma +(1-p)\delta )(1-\gamma -(1-p)\delta ) - p(1-p)\delta ^2[(1-p)(1-2\gamma )-2\delta (1-p)^2]}{[(\gamma +(1-p)\delta )(1-\gamma -(1-p)\delta )]^2} \end{aligned}$$

takes on the sign of its numerator.

Suppose $\delta \ge 0$. The numerator is positive if and only if

$$\begin{aligned} 2(\gamma +(1-p)\delta )(1-\gamma -(1-p)\delta )&> \delta ((1-p)(1-2\gamma )-2(1-p)^2 \delta ) \;\; \Leftrightarrow \\ 2\gamma (1-\gamma )+2\delta (1-p)- 4\gamma \delta (1-p)-2\delta ^2(1-p)^2&> \delta (1-p)-2\gamma \delta (1-p) -2\delta ^2(1-p)^2 \;\; \Leftrightarrow \\ 2\gamma (1-\gamma ) + \delta (1-p)&> 2\gamma \delta (1-p). \end{aligned}$$

(C1)

Recall that $\gamma +\delta \le 1$. Then

$$\begin{aligned} 2\gamma (1-\gamma )+\delta (1-p) \ge 2\gamma (1-\gamma ) > 2\gamma (1-\gamma )(1-p) \ge 2 \gamma \delta (1-p), \end{aligned}$$

and the middle inequality holds for $p \in (0,1)$. Thus, condition (C1) holds for $\delta \ge 0$.

Suppose $\delta <0$. The numerator is now negative if and only if condition (C1) holds. Rewrite (C1) as

$$\begin{aligned} 2\gamma (1-\gamma ) > \delta (1-p)[2\gamma - 1]. \end{aligned}$$

If $\gamma \ge 1/2$, the right-hand side is weakly negative, and the left-hand side is strictly positive, so the inequality holds. If $\gamma < 1/2$, then $(1-\gamma ) > (2\gamma - 1)$. Also, $2 > (1-p)$, and $\gamma \ge | \delta |$ because $\gamma + \delta \ge 0$. Thus, (C1) holds for $\delta <0$. $\square$

Proof of Result 2

From (5),

$$\begin{aligned} \frac{\displaystyle \partial \Omega (\delta ,\gamma ,p)}{\displaystyle \partial \gamma }=\frac{-p(1-p)\delta ^2(1-2\delta (1-p)-2\gamma )}{[\gamma (1-2\delta (1-p))-\gamma ^2 + \delta (1-p)(1-\delta (1-p))]^2}, \end{aligned}$$

and

$$\begin{aligned} \frac{\displaystyle \partial ^2 \Omega (\delta ,\gamma ,p)}{\displaystyle \partial \gamma ^2}&=\frac{2p(1-p)\delta ^2\left[ \gamma (1-2\delta (1-p))-\gamma ^2 + \delta (1-p)(1-\delta (1-p))\right] ^2}{\left[ \gamma (1-2\delta (1-p))-\gamma ^2 + \delta (1-p)(1-\delta (1-p))\right] ^4} \\&+\quad \frac{2p(1-p)\delta ^2(1-2\delta (1-p)-2\gamma )^2[\gamma (1-2\delta (1-p))-\gamma ^2 + \delta (1-p)(1-\delta (1-p))]}{[\gamma (1-2\delta (1-p))-\gamma ^2 + \delta (1-p)(1-\delta (1-p))]^4}, \end{aligned}$$

the latter of which is the sum of two positive terms. $\Omega (\delta ,\gamma ,p)$ is therefore convex in $\gamma$. Solving the first-order condition,

$$\begin{aligned} \mathrm{argmin}_{\gamma } \Omega (\delta ,\gamma ,p) = 1/2-\delta (1-p). \,\, \end{aligned}$$

$\square$

Proof of Result 3

First, rewrite (5) as

$$\begin{aligned} \Omega (\delta ,\gamma ,p) = \frac{\delta ^2(p-p^2)}{\gamma (1-\gamma )+\delta (1-\delta )-2\gamma \delta +\delta (2\delta +2\gamma -1)p-\delta ^2p^2}. \end{aligned}$$

Then

$$\begin{aligned} \frac{\displaystyle \partial \Omega (\delta ,\gamma ,p)}{\displaystyle \partial p}&=\frac{\delta ^2[\gamma (1-\gamma )+\delta (1-\delta )-2\gamma \delta +\delta (2\delta +2\gamma -1)p-\delta ^2p^2](1-2p)}{[\gamma (1-\gamma )+\delta (1-\delta )-2\gamma \delta +\delta (2\delta +2\gamma -1)p-\delta ^2p^2]^2} \\&\quad -\frac{\delta ^2\left( p-p^2\right) \left( \delta (2\delta +2\gamma -1)-2\delta ^2p\right) }{\left[ \gamma (1-\gamma )+\delta (1-\delta )-2\gamma \delta +\delta (2\delta +2\gamma -1)p-\delta ^2p^2\right] ^2} \\&=\frac{\delta ^2[(1-2p)\lambda + (1-\delta -2\gamma )\delta p^2]}{\left[ \gamma (1-\gamma )+\delta (1-\delta )-2\gamma \delta +\delta (2\delta +2\gamma -1)p-\delta ^2p^2\right] ^2}, \end{aligned}$$

where

$$\begin{aligned} \lambda \equiv \gamma (1-\gamma ) + \delta (1-\delta ) - 2 \gamma \delta . \end{aligned}$$

Further, define

$$\begin{aligned} \Psi \equiv \lambda +\delta (2\delta +2\gamma -1)p-\delta ^2p^2. \end{aligned}$$

Then

$$\begin{aligned} \frac{\displaystyle \partial ^2 \Omega (\delta ,\gamma ,p)}{\displaystyle \partial p^2}&= \frac{\delta ^2 \Psi ^2[-2\lambda +2\delta (1-\delta -2\gamma )p]}{\Psi ^4} \\&\quad - \frac{2\delta ^2 \Psi [(1-2p)\lambda + (1-\delta -2\gamma )\delta p^2][\delta (2\delta +2\gamma -1)-2\delta ^2p]}{\Psi ^4} \\&= \frac{2 \delta ^2 \left[ \lambda +\delta (2\delta +2\gamma -1)p-\delta ^2p^2\right] \left[ -\lambda +\delta (1-\delta -2\gamma )p\right] }{\Psi ^3} \\&\quad - \frac{2\delta ^2 [(1-2p)\lambda + (1-\delta -2\gamma )\delta p^2][\delta (2\delta +2\gamma -1)-2\delta ^2p]}{\Psi ^3} \\&= \frac{2\delta ^2 \lambda }{\Psi ^3} [ 3 \delta ^2 p(1-p) - \gamma (1-\gamma ) - \delta ^2], \end{aligned}$$

where the last equality follows after copious algebra.

Note that

$$\begin{aligned} \lambda \ge 0 \; \Leftrightarrow \; \delta (1-\delta ) + \gamma (1-\gamma ) \ge 2 \gamma \delta . \end{aligned}$$

For $\delta \ge 0$, $\gamma +\delta \le 1$, so $1-\gamma \ge \delta$ and $1-\delta \ge \gamma$. Then

$$\begin{aligned} \delta (1-\delta ) + \gamma (1-\gamma ) \ge \gamma \delta + \delta \gamma \ge 2 \delta \gamma . \end{aligned}$$

For $\delta < 0$,

$$\begin{aligned} \lambda \ge 0 \; \Leftrightarrow \; \gamma (1-\gamma ) + 2\gamma (-\delta ) \ge (-\delta )(1+(-\delta )). \end{aligned}$$

But $\delta < 0$ implies $\gamma +\delta \ge 0$, so $\gamma \ge -\delta$. Then

$$\begin{aligned} \gamma (1-\gamma ) + 2\gamma (-\delta ) \ge \gamma (1-\gamma ) -2\gamma ^2 = \gamma (1-\gamma ) \ge (-\delta )(1+(-\delta )). \end{aligned}$$

Thus, $\lambda >0$ unless $\gamma =|\delta |=1/2$. Recall that $\Psi =(1-\gamma _u)\gamma _u>0$. Therefore,

$$\begin{aligned} \frac{\displaystyle \partial ^2 \Omega (\delta ,\gamma ,p)}{\displaystyle \partial p^2}< 0 \; \Leftrightarrow \; 3p(1-p)<1+\frac{\gamma (1-\gamma )}{\delta ^2}. \end{aligned}$$

Because $p(1-p)$ is maximized at $p=1/2$,

$$\begin{aligned} 3p(1-p)< 3/4 < 1+\frac{\gamma (1-\gamma )}{\delta ^2}, \end{aligned}$$

so $\Omega (\delta ,\gamma ,p)$ is strictly concave in p.

Next, define

$$\begin{aligned} p^* \equiv \mathrm{argmax}_{p} \Omega (\delta ,\gamma ,p). \end{aligned}$$

If $\delta +2\gamma =1$, then $p^*=1/2$. Otherwise, the first-order condition yields

$$\begin{aligned} p^* = \frac{\lambda \pm \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}}{\delta (1-\delta -2\gamma )}. \end{aligned}$$

Recall $\lambda \ge 0$. The term under the square root is weakly positive if

$$\begin{aligned} \lambda&\ge \delta (1-\delta -2\gamma ) \;\, \Leftrightarrow \\ \gamma (1-\gamma )&\ge 0, \end{aligned}$$

which is true by assumption, so $p^*$ is a real number.

Suppose $\delta >0$. Then

$$\begin{aligned} \lambda&> \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )} \;\, \Leftrightarrow \\ 1&> \delta + 2\gamma . \end{aligned}$$

For $\delta + 2\gamma > 1$, the only positive solution to the quadratic formula is

$$\begin{aligned} p^* = \frac{\lambda - \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}}{\delta (1-\delta -2\gamma )}, \end{aligned}$$

which is strictly less than one if and only if

$$\begin{aligned} - \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}&> \delta (1-\delta -2\gamma ) - \lambda \;\, \Leftrightarrow \\ \lambda ^2 - \lambda \delta (1-\delta -2\gamma )&< \delta ^2 (1-\delta -2\gamma )^2 - 2 \lambda \delta (1-\delta -2\gamma ) + \lambda ^2 \;\, \Leftrightarrow \\ \lambda \delta (1-\delta -2\gamma )&< \delta ^2 (1-\delta -2\gamma )^2 \;\, \Leftrightarrow \\ \lambda&> \delta (1-\delta - 2\gamma ) \;\, \Leftrightarrow \\ \gamma (1-\gamma )&>0. \end{aligned}$$

Again, the last inequality holds by assumption.

For $\delta + 2 \gamma < 1$, both solutions to the quadratic formula are positive, but

$$\begin{aligned} p^* = \frac{\lambda + \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}}{\delta (1-\delta -2\gamma )}&> 1 \;\, \Leftrightarrow \\ \gamma (1-\gamma )&>0. \end{aligned}$$

The other solution is strictly less than one if and only if

$$\begin{aligned} - \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}&< \delta (1-\delta -2\gamma ) - \lambda \;\, \Leftrightarrow \\ \lambda ^2 - \lambda \delta (1-\delta -2\gamma )&> \delta ^2 (1-\delta -2\gamma )^2 - 2 \lambda \delta (1-\delta -2\gamma ) + \lambda ^2 \;\, \Leftrightarrow \\ \lambda \delta (1-\delta -2\gamma )&> \delta ^2 (1-\delta -2\gamma )^2 \;\, \Leftrightarrow \\ \gamma (1-\gamma )&>0. \end{aligned}$$

Thus, if $\delta > 0$ (and $\delta + 2 \gamma \ne 1$),

$$\begin{aligned} p^* = \frac{\lambda - \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )}}{\delta (1-\delta -2\gamma )} < 1. \end{aligned}$$

Now suppose $\delta < 0$. Then

$$\begin{aligned} \lambda&> \sqrt{\lambda ^2 - \lambda \delta (1-\delta -2\gamma )} \;\, \Leftrightarrow \\ 1&< \delta + 2\gamma . \end{aligned}$$

Similar arguments again yield a unique solution of

$$\begin{aligned} p^* = \frac{\lambda - \sqrt{\lambda ^2 - \lambda \delta \left( 1-\delta -2\gamma \right) }}{\delta (1-\delta -2\gamma )} < 1. \,\, \end{aligned}$$

$\square$

A.2 Power in the Case of Three Groups

Figures 5, 6, 7 display power as a function of the parameters in a hit rates test of three groups. A reference group has baseline hit rate $\gamma$. For groups one and two, the differences in hit rates from the baseline are denoted by $\delta _1$ and $\delta _2$, respectively. The proportion of searches in groups one and two are denoted by $p_1$ and $p_2$.

Fig. 5

Relationship between hit rate differences ($\delta _1$, $\delta _2$) and power ($\beta$) in a test of three groups Note: Power is calculated with sample size $n=1500$, significance criterion $\alpha =.05$, respective proportions $p_1=.6$ and $p_2=.1$ of searches in the first and second groups, and hit rate level $\gamma =0.3$

Fig. 6

Relationship between hit rate level ($\gamma$) and power ($\beta$) for varying hit rate differences ($\delta _1=\delta _2=\delta$) in a test of three groups Note: Power is calculated with sample size $n=1500$, significance criterion $\alpha =.05$, and respective proportions $p_1=.6$ and $p_2=.1$ of searches in the first and second groups

Fig. 7

Relationship between proportion of searches in first and second groups ($p_1$, $p_2$) and power ($\beta$) in a test of three groups Note: Power is calculated with sample size $n=1500$, significance criterion $\alpha =.05$, respective differences in hit rates of $\delta _1=.05$ and $\delta _2=.05$ for the first and second groups, and hit rate level $\gamma =.3$

A.4 Simulation with an Expanded Sample Size

See Fig. 8

Fig. 8

Simulation–proportion of p-values below significance levels (larger sample) Note: Simulations are based on 10, 000 replications, each with a sample size of $n=5000$. $\theta _b$ denotes the rate at which police use a more intensive search with black drivers. The respective rate for white drivers is $\theta _w=.10$. The proportion of white drivers in the search population is $p=.3$. The rate of contraband possession is $c=0.5$ for both groups. The police detect contraband with probability $d=.2$ with an ordinary search and raise the probability by $\phi =0.3$ with an intensive search. Police are slightly biased against black drivers, with $\eta =-.02$

A.5 The Hit Rates Model with a Resource Constraint on Intensive Searches

Consider the classical hit rates of model of Knowles et al. (2001) with a binary, intensive search technology. (The description here is somewhat informal.) For a continuum of police officers and motorists, let $r \in \{B,W\}$ denote the race of the motorist, and let $c \in \mathbb {R}$ denote other characteristics an officer may use in the decision to search a vehicle. The notation for c matches that of the classical model and should not be confused with the crime rate in the discussion above. Race is everywhere observable. The officer observes c, but the econometrician may not observe c, which follows distribution function F(c|r).

Officers choose whether to search a motorist of type (c, r). They have access to a binary search technology, $b \in \{0,1\}$. If they search a guilty motorist without the technology, they detect the crime with probability d. With the technology, they raise the probability of detection to $d+\phi$. Officers maximize total convictions subject to the cost of search. The marginal cost of searching a motorist of race r without the technology is $t_r$. Police discrimination operates when $t_B \ne t_W$. An arrest provides a normalized benefit equal to one, so cost is scaled as a fraction of the benefit. Assume $t_r \le d$, i.e., a search would always be worth conducting if every motorist carries contraband.

Motorists decide whether to carry contraband. If they do not carry, they obtain a zero payoff. If they do carry, they obtain $-j(c,r)$ if the crime is detected and v(c, r) if the crime is not detected. Both j(c, r) and v(c, r) are strictly positive.

Let $\mu ^b(c,r)$ denote the probability an officer searches a motorist of type (c, r) with technology b. The expected payoff to contraband is

$$\begin{aligned} \left[ \mu ^0(c,r)d+\mu ^1(c,r)(d+\phi )\right] \left[ -j(c,r)\right] +\left[ 1-\mu ^0(c,r)d-\mu ^1(c,r)(d+\phi )\right] \left[ v(c,r)\right] . \end{aligned}$$

A motorist will carry contraband if and only if the expected payoff is greater than zero. If the payoff is exactly zero, motorists are indifferent and may randomize over whether to carry. Denote the probability a motorist of type (c, r) is guilty by P(G|c, r).

Officers choose search probabilities to solve

$$\begin{aligned} \max _{\begin{array}{c} \mu ^0(c,B),\mu ^1(c,B)\\ \mu ^0(c,W),\mu ^1(c,W) \end{array}} \sum _{r \in \{B,W\}}&\int [d P(G|c,r) - t_r][ \mu ^0(c,r)] + (d+\phi ) P(G|c,r) \mu ^1(c,r) f(c|r) dc \\ \text {s.t.}&\int \mu ^1(c,B) f(c|B) + \mu ^1(c,W) f(c|W) dc \le M \\&\mu ^0(c,r) + \mu ^1(c,r) \le 1, \end{aligned}$$

where M denotes a resource constraint on the use of intensive searches. In any equilibrium with a positive crime rate, the resource constraint will bind.

For a motorist to optimally randomize,

$$\begin{aligned} \frac{v(c,r)}{v(c,r)+j(c,r)} = \mu ^0(c,r)d+\mu ^1(c,r)(d+\phi ). \end{aligned}$$

Define

$$\begin{aligned} u(c,r) \equiv \frac{v(c,r)}{v(c,r)+j(c,r)}. \end{aligned}$$

If the same characteristics have the same ordinal relationship to u(c, r) for both races, then reorder c so u(c, r) is increasing. Assume

$$\begin{aligned} \lim _{c \rightarrow - \infty } u(c,r) = 0, \; \text {and} \lim _{c \rightarrow \infty } u(c,r) = 1. \end{aligned}$$

Define $\bar{c}_r$ as the solution to

$$\begin{aligned} \frac{v(c,r)}{v(c,r)+j(c,r)} = d + \phi . \end{aligned}$$

Let u(c, r) follow distribution function H(c, r). If $H(\bar{c}_r,r)<1$, then some motorists of race r will not be deterred even when $\mu ^1(c,r)=1$.

First, suppose $H(\bar{c}_r,r)=1$ for both races. Suppose officers are unbiased, so $t_B=t_W=t$. If $P^*(G|c,r)>t/d$, then $\mu ^0(c,r)+\mu ^1(c,r)=1$, and the randomization condition for motorists is violated. If $P^*(G|c,r)<t/d$, then $\mu ^0(c,r)=0$ and $\mu ^1(c,r)=u(c,r)/(d+\phi )$ for all (c, r), but the latter equality is negated by a sufficiently binding resource constraint. Furthermore, an equilibrium of exclusively intensive searches would be incompatible with observational data. Thus, $P^*(G|c,r)=t/d$.

The equality of crime rates in equilibrium provides the basis for a test of police bias. Hit rates for each race are

$$\begin{aligned} D(r) = \int P^*(G|c,r) \frac{\mu ^{*0}(c,r) d f(c|r)}{\int \mu ^{*0}(c,r) f(s|r) ds} + P^*(G|c,r) \frac{\mu ^{*1}(c,r) (d + \phi ) f(c|r)}{\int \mu ^{*1}(c,r) f(s|r) ds} dc. \end{aligned}$$

Because $P^*(G|c,r)=t/d$ implies $D(W)=D(B)$, the standard hit rates test applies irrespective of the search technology.

Now suppose $H(\bar{c}_r,r)<1$ for at least one race. Police form correct beliefs on H(c, r) in equilibrium. For simplicity, take the case of

$$\begin{aligned} 1 - H(\bar{c}_B,B) + 1 - H(\bar{c}_W,W) \ge M. \end{aligned}$$

Up to the resource constraint, police intensively search motorists of each race with probability one when $c \ge \bar{c}_r$. The division of intensive searches across race is arbitrary because the resource constraint binds before police enter the domain of deterrable motorists. (If the inequality did not hold, police would first intensively search non-deterrable motorists with probability one before randomizing with the leftover intensive search capacity as above.) For the remainder of motorists, equilibrium is analogous to the case of complete deterrability, and hit rates are

$$\begin{aligned} D(r) = \int _{c < \bar{c}_r} P^*(G|c,r) \frac{\mu ^{*0}(c,r) d f(c|r)}{\int \mu ^{*0}(c,r) f(s|r) ds} dc + \int _{c \ge \bar{c}_r} (d + \phi ) dc, \end{aligned}$$

where, again, $P^*(G|c,r)=t/d$. Then

$$\begin{aligned} D(B) = D(W) + \int _{c \ge \bar{c}_B} (d + \phi ) dc - \int _{c \ge \bar{c}_W} (d + \phi ) dc. \end{aligned}$$

The weighted difference in intensive searches should thus be subtracted from the difference in hit rates prior to any hit rates test. Because crime rates are equal in equilibrium, the subtraction is everywhere valid. It is necessary unless all crime is deterrable.

The adjusted difference in hit rates therefore offers a more accurate test of police bias when the econometrician does not observe all characteristics of every stop and search. The empirical adjustment for search intensity above can be calculated in any data set with a binary search technology. Lastly, when $1 - H(\bar{c}_B,B) + 1 - H(\bar{c}_W,W) < M$, the adjustment is necessary only for motorists of type $c \ge \bar{c}_r$, but the unobservability of that threshold is unproblematic since the adjustment is valid for all c.