Skip to main content
SpringerLink
Log in

Kullback–Leibler divergence for Bayesian nonparametric model checking

  • Research Article
  • Published:
Journal of the Korean Statistical Society Aims and scope Submit manuscript

Abstract

Bayesian nonparametric statistics is an area of considerable research interest. While recently there has been an extensive concentration in developing Bayesian nonparametric procedures for model checking, the use of the Dirichlet process, in its simplest form, along with the Kullback–Leibler divergence is still an open problem. This is mainly attributed to the discreteness property of the Dirichlet process and that the Kullback–Leibler divergence between any discrete distribution and any continuous distribution is infinity. The approach proposed in this paper, which is based on incorporating the Dirichlet process, the Kullback–Leibler divergence and the relative belief ratio, is considered the first concrete solution to this issue. Applying the approach is simple and does not require obtaining a closed form of the relative belief ratio. A Monte Carlo study and real data examples show that the developed approach exhibits excellent performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abramowitz, M., & Stegun, I. A. (1972). Handbook of mathematical functions with formulas, graphs, and mathematical tables. New York: Dover.

    MATH  Google Scholar 

  • Alizadeh Noughabi, H. (2010). A new estimator of entropy and its application in testing normality. Journal of Statistical Computation and Simulation, 80, 1151–1162.

    MathSciNet  MATH  Google Scholar 

  • Alizadeh Noughabi, H., & Arghami, N. R. (2010). A new estimator of entropy. Journal of the Iranian Statistical Society, 9, 53–64.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L. (2018). The two-sample problem via relative belief ratio. https://arxiv.org/abs/1805.07238.

  • Al-Labadi, L., & Abdelrazeq, I. (2017). On functional central limit theorems of Bayesian nonparametric priors. Statistical Methods and Applications, 26, 215–229.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Evans, M. (2017). Optimal robustness results for relative belief inferences and the relationship to prior-data conflict. Bayesian Analysis, 12, 705–728.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Evans, M. (2018). Prior-based model checking. Canadian Journal of Statistics, 46, 380–398.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., Patel, V., Vakiloroayaei, K., & Wan, C.(2018). A Bayesian nonparametric estimation to entropy. https://arxiv.org/abs/1903.00655.

  • Al-Labadi, L., & Zarepour, M. (2013a). A Bayesian nonparametric goodness of fit test for right censored data based on approximate samples from the beta-Stacy process. Canadian Journal of Statistics, 41, 466–487.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Zarepour, M. (2013b). On asymptotic properties and almost sure approximation of the normalized inverse-Gaussian process. Bayesian Analysis, 8, 553–568.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Zarepour, M. (2014a). Goodness of fit tests based on the distance between the Dirichlet process and its base measure. Journal of Nonparametric Statistics, 26, 341–357.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Zarepour, M. (2014b). On simulations from the two-parameter Poisson–Dirichlet process and the normalized inverse-Gaussian process. Sankhyā A, 76, 158–176.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., & Zarepour, M. (2017). Two-sample Kolmogorov–Smirnov test using a Bayesian nonparametric approach. Mathematical Methods of Statistics, 26, 212–225.

    MathSciNet  MATH  Google Scholar 

  • Al-Labadi, L., Zeynep, B., & Evans, M. (2017). Goodness of fit for the logistic regression model using relative belief. Journal of Statistical Distributions and Applications,. https://doi.org/10.1186/s40488-017-0070-7.

    Article  MATH  Google Scholar 

  • Al-Labadi, L., Zeynep, B., & Evans, M. (2018). Statistical reasoning: Choosing and checking the ingredients, inferences based on a measure of statistical evidence with some applications. Entropy, 20, 289. https://doi.org/10.3390/e20040289.

    Article  Google Scholar 

  • Al-Omari, A. I. (2014). Estimation of entropy using random sampling. Journal of Computation and Applied Mathematics, 261, 95–102.

    MathSciNet  MATH  Google Scholar 

  • Al-Omari, A. I. (2016). A new measure of entropy of continuous random variable. Journal of Statistical Theory and Practice, 10, 721–735.

    MathSciNet  MATH  Google Scholar 

  • Andrews, D. F., & Herzberg, A. M. (1985). Data–A collection of problems from many fields for the student and research worker. Berlin: Springer.

    MATH  Google Scholar 

  • Baskurt, Z., & Evans, M. (2013). Hypothesis assessment and inequalities for Bayes factors and relative belief ratios. Bayesian Analysis, 8, 569–590.

    MathSciNet  MATH  Google Scholar 

  • Berger, J. O., & Guglielmi, A. (2001). Bayesian testing of a parametric model versus nonparametric alternatives. Journal of the American Statistical Association, 96, 174–184.

    MathSciNet  MATH  Google Scholar 

  • Bondesson, L. (1982). On simulation from infinitely divisible distributions. Advances in Applied Probability, 14, 885–869.

    MathSciNet  MATH  Google Scholar 

  • Bouzebda, S., Elhattab, I., Keziou, A., & Lounis, T. (2013). New entropy estimator with an application to test of normality. Communications in Statistics - Theory and Methods, 42, 2245–2270.

    MathSciNet  MATH  Google Scholar 

  • Carota, C., & Parmigiani, G. (1996). On Bayes factors for nonparametric alternatives. In J. M. Bernardo, J. Berger, A. P. Dawid, & A. F. M. Smith (Eds.), Bayesian Statistics 5. London: Oxford University Press.

    Google Scholar 

  • Correa, J. C. (1995). A new estimator of entropy. Communications in Statistics—Theory and Methods, 24, 2439–2449.

    MathSciNet  MATH  Google Scholar 

  • Cover, T. M., & Thomas, J. A. (1991). Elements of information theory (2nd ed.). New York: Wiley.

    MATH  Google Scholar 

  • Ebrahimi, N., Pflughoeft, K., & Soofi, E. (1994). Two measures of sample entropy. Statistics and Probability Letters, 20, 225–234.

    MathSciNet  MATH  Google Scholar 

  • Evans, M. (2015). Measuring Statistical Evidence Using Relative Belief, Monographs on Statistics and Applied Probability (Vol. 144). Boca Raton: CRC Press, Taylor and Francis Group.

    Google Scholar 

  • Evans, M., & Moshonov, H. (2006). Checking for prior-data conflict. Bayesian Analysis, 1, 893–914.

    MathSciNet  MATH  Google Scholar 

  • Evans, M., & Swartz, T. (1994). Distribution theory and inference for polynomial-normal densities. Communications in Statistics—Theory and Methods, 23, 1123–1148.

    MathSciNet  MATH  Google Scholar 

  • Evans, M., & Tomal, J. (2018). Measuring statistical evidence and multiple testing. FACET, 3, 563–583.

    Google Scholar 

  • Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1, 209–230.

    MathSciNet  MATH  Google Scholar 

  • Florens, J. P., Richard, J. F., & Rolin, J. M. (1996). Bayesian encompassing specification tests of a parametric model against a nonparametric alternative. Technical Report 9608, Universitsé Catholique de Louvain, Institut de statistique.

  • Grzegorzewski, P., & Wieczorkowski, R. (1999). Entropy-based goodness-of-fittest for exponentiality. Communications in Statistics—Theory and Methods, 28, 1183–1202.

    MathSciNet  MATH  Google Scholar 

  • Holmes, C. C., Caron, F., Griffin, J. E., & Stephens, D. A. (2015). Two-sample Bayesian nonparametric hypothesis testing. Bayesian Analysis, 2, 297–320.

    MathSciNet  MATH  Google Scholar 

  • Hsieh, P. (2011). A nonparametric assessment of model adequacy based on Kullback–Leibler divergence. Statistics and Computing, 23, 149–162.

    MathSciNet  MATH  Google Scholar 

  • Ishwaran, H., & James, L. F. (2001). Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 96, 161–173.

    MathSciNet  MATH  Google Scholar 

  • Ishwaran, H., & Zarepour, M. (2002). Exact and approximate sum representations for the Dirichlet process. Canadian Journal of Statistics, 30, 269–283.

    MathSciNet  MATH  Google Scholar 

  • Jordan, M. I. (2011). What are the open problems in Bayesian statistics? ISBA Bulletin, 18, 1–4.

    Google Scholar 

  • McVinish, R., Rousseau, J., & Mengersen, K. (2009). Bayesian goodness of fit testing with mixtures of triangular distributions. Scandivavian Journal of Statistics, 36, 337–354.

    MathSciNet  MATH  Google Scholar 

  • Noughabi, H. A., & Arghami, N. R. (2013). General treatment of goodness-of-fit tests based on Kullback–Leibler information. Journal of Statistical Computation and Simulation, 83, 1556–1569.

    MathSciNet  MATH  Google Scholar 

  • Pérez-Rodríguez, P., Vaquera-Huerta, H., & Villaseñor-Alva, J. A. (2009). A goodness-of-fit test for the Gumbel distribution based on Kullback–Leibler information. Communications in Statistics: Theory and Methods, 38, 842–855.

    MathSciNet  MATH  Google Scholar 

  • Rudin, W. (1974). Real and Complex Analysis (2nd ed.). New York: McGrawHill.

    MATH  Google Scholar 

  • Sethuraman, J. (1994). A constructive definition of Dirichlet priors. Statistica Sinica, 4, 639–650.

    MathSciNet  MATH  Google Scholar 

  • Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(379–423), 623–656.

    MathSciNet  MATH  Google Scholar 

  • Swartz, T. B. (1999). Nonparametric goodness-of-fit. Communications in Statistics: Theory and Methods, 28, 2821–2841.

    MathSciNet  MATH  Google Scholar 

  • van Es, B. (1992). Estimating functionals related to a density by a class of statistics based on spacings. Scandinavian Journal of Statistics, 19, 61–72.

    MathSciNet  MATH  Google Scholar 

  • Vasicek, O. (1976). A test for normality based on sample entropy. Journal of Royal Statistical Society B, 38, 54–59.

    MathSciNet  MATH  Google Scholar 

  • Verdinelli, I., & Wasserman, L. (1998). Bayesian goodness-of-fit testing using finite-dimensional exponential families. Annals of Statistics, 26, 1215–1241.

    MathSciNet  MATH  Google Scholar 

  • Viele, K. (2007). Nonparametric estimation of Kullback–Leibler information illustrated by evaluating goodness of fit. Bayesian Analysis, 2, 239–280.

    MathSciNet  MATH  Google Scholar 

  • Wieczorkowski, R., & Grzegorzewski, P. (1999). Entropy estimators-improvements and comparisons. Communications in Statistics—Simulation and Computation, 28, 541–567.

    MathSciNet  MATH  Google Scholar 

  • Wolpert, R. L., & Ickstadt, K. (1998). Simulation of Lévy random fields. In D. Day, P. Muller, & D. Sinha (Eds.), Practical nonparametric and semiparametric Bayesian statistics (pp. 227–242). Berlin: Springer.

    Google Scholar 

  • Zarepour, M., & Al-Labadi, L. (2012). On a rapid simulation of the Dirichlet process. Statistics and Probability Letters, 82, 916–924.

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematical and Computational Sciences, University of Toronto Mississauga, Mississauga, ON, L5L 1C6, Canada

    Luai Al-Labadi, Vishakh Patel, Kasra Vakiloroayaei & Clement Wan

Authors
  1. Luai Al-Labadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vishakh Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kasra Vakiloroayaei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Clement Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luai Al-Labadi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Labadi, L., Patel, V., Vakiloroayaei, K. et al. Kullback–Leibler divergence for Bayesian nonparametric model checking. J. Korean Stat. Soc. 50, 272–289 (2021). https://doi.org/10.1007/s42952-020-00072-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42952-020-00072-7

Keywords

Mathematics Subject Classification

Search

Navigation