Skip to main content

Advertisement

Log in

A moment inequality approach to statistical inference for rankings

  • Article
  • Published:
The Japanese Economic Review Aims and scope Submit manuscript

Abstract

This paper proposes an econometric method to construct confidence sets for rankings by considering the situations in which data on a comparison between each pair of alternatives are available. The key idea is that if an alternative, say A, is ranked higher than some other alternative, say B, then it would be natural to assume that drawing an observation in which A is compared favorably with B is more likely than the opposite. This provides an insight that we can use to test for a particular ranking by translating it into a problem of testing moment inequalities and that we can obtain confidence sets for rankings by inverting these tests of moment inequalities. We apply the proposed method to statistical inference concerning the rankings of teams in the Nippon Professional Baseball league in Japan.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. See, for the example, “University Rankings Go Global” New York Times, April 14, 2014, https://www.nytimes.com/2013/04/14/education/edlife/university-rankings-go-global.html.

  2. https://www.japanuniversityrankings.jp/topics/00154/index.html.

  3. If a sample of individual rankings is available, \(D_{ij}\) can be constructed in the following way. For each individual ranking, let \(D_{ij}=1\) if i is ranked higher than j in that ranking. For example, university rankings are typically constructed by aggregating various indexes which may be considered as individual rankings. We can thus convert the data on indexes into data on pairwise comparisons between two universities. Namely, if university i ranks higher than j in the nth index, we regard that the nth observation of \(D_{ij}\) is one.

  4. Strictly speaking, a confidence set can be empty. However, the probability of the null set, \(\emptyset\), including the true ranking, is obviously zero. The proposed procedure never yields \(\emptyset\) as a confidence set in cases with two alternatives. If there are more than three alternatives, and transitivity is imposed, it is possible to obtain an empty confidence set (in such a case, transitivity is rejected). If willing to drop transitivity as an assumption, the proposed method never yields an empty confidence set.

  5. If correlation exists among observations, the t test statistic should be modified such that the denominator is a correlation-robust standard error. The remainder of the procedure does not need alternation.

  6. As explained in footnote 3, if a sample of individual rankings is available, it can be converted into a sample of pairwise comparisons. The number of indexes corresponds to \(N=N_{ij}\) in our setting. If index n ranks alternative i over j, then we set \(D_{nij}=1\). We then compute \({\hat{p}}_{ij} = \sum _{n=1}^N D_{nij} /N\). \({\hat{p}}_{ij}\) is the fraction of indexes that rank i higher than j.

  7. See, e.g., “The 27th survey of popular sports” conducted by Chuo Chousa-sha in April 2017 available at https://www.crs.or.jp/data/pdf/sports19.pdf.

  8. See, e.g., “Softbank nihon-ichi nara, keizaikouka 313 oku yen (Softbank’s winning the Japan series would yield an economic impact of 31,300,000,000 JPY),” Nishinippon Shimbun, October 28, 2018. https://www.nishinippon.co.jp/item/n/460934/. Headline translation by the author. The Japan Series is the most prestigious league in the NPB, where after a regular season each league holds a final series, known in Japan as the Climax Series, played by the three highest-ranked teams. The winners of the Climax Series from each league then play the Japan Series.

References

  • Alvo, M., & Yu, P. L. H. (2014). Statistical methods for ranking data. Berlin: Springer.

    Book  Google Scholar 

  • Bontemps, C., & Magnac, T. (2017). Set identification, moment restrictions, and inference. Annual Review of Economics, 9, 103–129.

    Article  Google Scholar 

  • Canay, Ivan A., & Shaikh, Azeem M. (2017). Practical and theoretical advances in inference for partially identified models. In B. Honoré, A. Pakes, M. Piazzesi, & L. Samuelson (Eds.), Advances in economics and econometrics: Eleventh world congress. Econometric society monographs (Vol. 2, pp. 271–306). Cambridge: Cambridge University Press. https://doi.org/10.1017/9781108227223.009.

    Chapter  Google Scholar 

  • Chernozhukov, V., Chetverikov, D., & Kato, K. (2019). Inference on causal and structural parameters using many moment inequalities. The Review of Economic Studies, 86(5), 1867–1900.

    Article  Google Scholar 

  • Ciliberto, F., & Tamer, E. (2009). Market structure and multiple equilibria in airline markets. Econometrica, 77(6), 1791–1828.

    Article  Google Scholar 

  • Dzemski, A., & Okui, R. (2020). Confidence set for group membership. mimeo. Available at https://arxiv.org/abs/1801.00332

  • Hall, P., & Miller, H. (2010). Modeling the variability of rankings. The Annals of Statistics, 38, 2652–2677.

    Google Scholar 

  • Hall, P., & Schimek, M. G. (2012). Moderate-deviation-based inference for random degeneration in paired rank lists. Journal of the American Statistical Association, 107(498), 661–672.

    Article  Google Scholar 

  • Jin, Y., & Okui, R. (2020). Testing for overconfidence statistically: A moment inequality approach. Journal of Applied Econometrics (forthcoming). https://doi.org/10.1002/jae.2780

  • Kemeny, J. G. (1959). Mathematics without numbers. Daedalus, 88(4), 577–591.

    Google Scholar 

  • Molinari, F. (2020). Microeconometrics with partial identification. In: Hand-book of econometrics, vol. 7A. Elsevier (forthcoming). Available at https://arxiv.org/abs/2004.11751

  • Romano, J. P., Shaikh, A. M., & Wolf, M. (2014). A practical two-step method for testing moment inequalities. Econometrica, 82(5), 1979–2002.

    Article  Google Scholar 

  • Sakai, T. (2013). Syakai Sentaku Riron heno Syoutai (Invitation to social choice theory). in Japanese. Nihon Hyohron Sha.

  • Stock, J. H., & Watson, M. W. (2001). Vector autoregressions. The Journal of Economic Perspectives, 15(4), 101–115.

    Article  Google Scholar 

  • Young, H. P. (1988). Condorcet’s theory of voting. American Political science review, 82(4), 1231–1244.

    Article  Google Scholar 

Download references

Acknowledgements

The author would like to thank Hiroshi Teruyama, an anonymous referee, Mototsugu Shintani, Chiaki Hara, Yukihiko Funaki, Naoya Sueishi, Maki Michinaka and participants at the 2020 Spring Meeting of the Japanese Economic Association for their helpful comments. Thanks also to KIM Hyeongyu for excellent research assistance with data collection and cleaning. The author gratefully acknowledges the financial support of the School of Social Sciences and a New Faculty Startup Grant at Seoul National University and from the Housing and Commercial Bank Economic Research Fund in the Institute of Economic Research at Seoul National University. The usual disclaimer applies.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ryo Okui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Proof of Theorem 2

Appendix A: Proof of Theorem 2

The proof closely follows Chernozhukov et al. (2019, Proof of Theorem 4.2). Let \(c (x) = \Phi ^{-1} (1-x)\). Note that \(c_{\beta } = c(\beta / S_N)\). Define \({\hat{J}} = \{(i,j) \in \succ _T : t_{ij} \ge - 2c (\beta /S_N ) \}\) and

$$\begin{aligned} J = \left\{ (i,j) \in \succ _T : \frac{\sqrt{N_{ij}} (0.5-p_{ij} )}{\sqrt{p_{ij}(1-p_{ij})}} \ge - c \left( \frac{\beta }{S_N} \right) \right\} . \end{aligned}$$

Recall that \(E (A_{nij}) = 0.5-p_{ij}\) and the variance of \(A_{nij}\) is \(p_{ij}(1-p_{ij})\). Note that \(\succ _T\) is included in the confidence set with moment selection if \(\max _{(i,j) \in \succ _T} t_{ij} < c( (\alpha -2\beta )/ |\hat{J} | )\). The proof proceeds with three steps.

First, we show that

$$\begin{aligned} \lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr \left( \max _{(i,j) \in \succ _T \backslash J} \sum _{n=1}^{N_{ij}} A_{nij} \le 0 \right) \ge 1- \beta . \end{aligned}$$

The key observation is that \(\sum _{n=1}^{N_{ij}} A_{nij} > 0\) for some \((i,j) \in \succ _T \backslash J\) implies that

$$\begin{aligned} \max _{(i,j) \in \succ _T} \frac{N_{ij}^{-1/2} \sum _{n=1}^{N_{ij}} (A_{nij} -( 0.5 - p_{ij})}{\sqrt{p_{ij}(1-p_{ij})}} > c\left( \frac{\beta }{S_N}\right) . \end{aligned}$$

It holds by the Bonferroni inequality and the central limit theorem that

$$\begin{aligned}&\lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr \left( \max _{(i,j) \in \succ _T} \frac{N_{ij}^{-1/2} \sum _{n=1}^{N_{ij}} (A_{nij} -( 0.5 - p_{ij}))}{\sqrt{p_{ij}(1-p_{ij})}}> c\left( \frac{\beta }{S_N}\right) \right) \\&\quad \le \lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \sum _{(i,j) \in \succ _T} \Pr \left( \frac{N_{ij}^{-1/2} \sum _{n=1}^{N_{ij}} (A_{nij} -( 0.5 - p_{ij}))}{\sqrt{p_{ij}(1-p_{ij})}} > c\left( \frac{\beta }{S_N}\right) \right) \\&\quad = \sum _{(i,j) \in \succ _T} \frac{\beta }{S_N} = \beta . \end{aligned}$$

This completes the first step.

Second, we show that \(\lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr \left( {\hat{J}} \supset J \right) \ge 1- \beta\). Note that

$$\begin{aligned} \Pr \left( {\hat{J}} \not \supset J \right) \le \Pr \left( \max _{(i,j) \in \succ _T} \left( \frac{1}{\sqrt{N_{ij}}} \sum _{i=1}^{N_{ij}} (0.5 - p_{ij} - A_{nij}) - (2{\hat{\sigma }}_{ij} - \sigma _{ij} ) c\left( \frac{\beta }{S_N} \right) \right) > 0 \right) , \end{aligned}$$

where \({\hat{\sigma }}_{ij}^2 = \sum _{n=1}^{N_{ij}} (A_{nij} - {\bar{A}}_{ij})^2 / N_{ij}\) and \(\sigma _{ij} = p_{ij} (1- p_{ij})\). If \((1 - \sigma _{ij}/ {\hat{\sigma }}_{ij} ) \ge -r\) for some \(0< r < 1\), it holds

$$\begin{aligned} 2{\hat{\sigma }}_{ij} - \sigma _{ij} = {\hat{\sigma }}_{ij} \left( 1+ \left( 1- \frac{\sigma _{ij}}{{\hat{\sigma }}_{ij}} \right) \right) \ge {\hat{\sigma }}_{ij} (1-r). \end{aligned}$$

Thus, it follows that

$$\begin{aligned}&\Pr \left( \max _{(i,j) \in \succ _T} \left( \frac{1}{\sqrt{N_{ij}}} \sum _{i=1}^{N_{ij}} (0.5 - p_{ij} - A_{nij}) - (2{\hat{\sigma }}_{ij} - \sigma _{ij} ) c\left( \frac{\beta }{S_N} \right) \right)> 0 \right) \\&\quad \le \Pr \left( \max _{(i,j) \in \succ _T} \frac{N_{ij}^{-1/2} \sum _{n=1}^{N_{ij}} ( 0.5 - p_{ij} - A_{nij} ) }{{\hat{\sigma }}_{ij}}> (1-r) c\left( \frac{\beta }{S_N} \right) \right) \\&\qquad + \Pr \left( \max _{(i,j) \in \succ _T} \left| \frac{\sigma _{ij}}{{\hat{\sigma }}_{ij}} - 1 \right| > r\right) , \end{aligned}$$

for all \(0< r <1\). We take \(r = \min _{(i,j) \in \succ _T} N_{ij}^{-1/3}\) so that the second term on the right hand side of the above inequality vanishes. By the Bonferroni inequality and the standard asymptotic argument for t statistic, we have

$$\begin{aligned} \lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)}\Pr \left( \max _{(i,j) \in \succ _T} \frac{N_{ij}^{-1/2} \sum _{n=1}^{N_{ij}} ( 0.5 - p_{ij} - A_{nij} ) }{{\hat{\sigma }}_{ij}} > (1-r) c\left( \frac{\beta }{S_N} \right) \right) \le \beta . \end{aligned}$$

We thus complete the second step.

Lastly, we show the result of the theorem. Note that when \(|J|=0\), the limit of \(\Pr ( \succ _T \in C_{\alpha }^M)\) (\(\ge \Pr ( \max _{(i,j) \in \succ _T} t_{ij} <0)\)) is more than \(1-\alpha\) as shown in the first step and \(\beta < \alpha\) by definition. We consider cases with \(|J| \ge 1\). We observe that

$$\begin{aligned} \left\{ \max _{(i,j) \in \succ _T} t_{ij}> c\left( \frac{\alpha -2\beta }{S_R} \right) \right\} \cap \left\{ \max _{(i,j) \in \succ _T \backslash J} \sum _{n=1}^{N_{ij}} A_{nij} \le 0 \right\} \subset \left\{ \max _{(i,j) \in J} t_{ij} > c\left( \frac{\alpha -2\beta }{S_R} \right) \right\} . \end{aligned}$$

It also holds that

$$\begin{aligned} \left\{ \max _{(i,j) \in J} t_{ij}> c\left( \frac{\alpha -2\beta }{S_R} \right) \right\} \cap \left\{ {\hat{J}} \supset J \right\} \subset \left\{ \max _{(i,j) \in J} t_{ij} > c\left( \frac{\alpha -2\beta }{|J|} \right) \right\} \end{aligned}$$

because \(|J| < S_R\) if \({\hat{J}} \supset J\). Using the results obtained in the first and second steps yields

$$\begin{aligned} \lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr (\succ _T \notin C_{\alpha }^M) =&\lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr \left( \left\{ \max _{(i,j) \in \succ _T} t_{ij}> c \left( \frac{\alpha -2\beta }{S_R} \right) \right\} \right) \\ \le&\lim _{N_{ij} \rightarrow \infty \text { for all } (i,j)} \Pr \left( \max _{(i,j) \in J} t_{ij} > c\left( \frac{\alpha -2\beta }{|J|} \right) \right) + 2 \beta \\ \le&\sum _{(i,j) \in J} \frac{\alpha -2\beta }{|J|} + 2 \beta = \alpha . \end{aligned}$$

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Okui, R. A moment inequality approach to statistical inference for rankings. JER 72, 169–184 (2021). https://doi.org/10.1007/s42973-020-00068-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42973-020-00068-2

Keywords

JEL Classification

Navigation