Applications of Mathematics, Vol. 65, No. 3, pp. 311-330, 2020
Notion of information and independent component analysis
Una Radojičić, Klaus Nordhausen, Hannu Oja
Received November 29, 2019. Published online May 25, 2020.
Abstract: Partial orderings and measures of information for continuous univariate random variables with special roles of Gaussian and uniform distributions are discussed. The information measures and measures of non-Gaussianity including the third and fourth cumulants are generally used as projection indices in the projection pursuit approach for the independent component analysis. The connections between information, non-Gaussianity and statistical independence in the context of independent component analysis is discussed in detail.
References: [1] A. R. Barron: Entropy and the central limit theorem. Ann. Probab. 14 (1986), 336-342. DOI 10.1214/aop/1176992632 | MR 0815975 | Zbl 0599.60024
[2] A. J. Bell, T. J. Sejnowski: An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7 (1995), 1129-1159. DOI 10.1162/neco.1995.7.6.1129
[3] P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. II: Location. Ann. Stat. 3 (1975), 1045-1069. DOI 10.1214/aos/1176343240 | MR 0395021 | Zbl 0321.62055
[4] P. J. Bickel, E. L. Lehmann: Descriptive statistics for nonparametric models. III: Dispersion. Ann. Stat. 4 (1976), 1139-1158. DOI 10.1214/aos/1176343648 | MR 474620 | Zbl 0351.62031
[5] M. Bilodeau, D. Brenner: Theory of Multivariate Statistics. Springer Texts in Statistics, Springer, New York (1999). DOI 10.1007/b97615 | MR 1705291 | Zbl 0930.62054
[6] H. Chernoff, I. R. Savage: Asymptotic normality and efficiency of certain nonparametric test statistics. Ann. Math. Stat. 29 (1958), 972-994. DOI 10.1214/aoms/1177706436 | MR 0100322 | Zbl 0092.36501
[7] T. M. Cover, J. A. Thomas: Elements of Information Theory. Wiley Series in Telecommunications, John Wiley & Sons, New York (1991). DOI 10.1002/0471200611 | MR 1122806 | Zbl 0762.94001
[8] L. Faivishevsky, J. Goldberger: ICA based on a smooth estimation of the differential entropy. NIPS'08: Proceedings of the 21st International Conference on Neural Information Processing Systems. Curran Associates, New York, 2008, 433-440.
[9] J. L. Hodges, Jr., E. L. Lehmann: The efficiency of some nonparametric competitors of the $t$-test. Ann. Math. Stat. 27 (1956), 324-335. DOI 10.1214/aoms/1177728261 | MR 0079383 | Zbl 0075.29206
[10] P. J. Huber: Projection pursuit. Ann. Stat. 13 (1985), 435-475. DOI 10.1214/aos/1176349519 | MR 0790553 | Zbl 0595.62059
[11] A. Hyvärinen: New approximations of differential entropy for independent component analysis and projection pursuit. NIPS '97: Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10. MIT Press, Cambridge, 1998, 273-279.
[12] A. Hyvärinen, J. Karhunen, E. Oja: Independent Component Analysis. John Wiley & Sons, New York (2001). DOI 10.1002/0471221317
[13] M. C. Jones, R. Sibson: What is projection pursuit? J. R. Stat. Soc., Ser. A 150 (1987), 1-36. DOI 10.2307/2981662 | MR 0887823 | Zbl 0632.62059
[14] K. Kim, G. Shevlyakov: Why Gaussianity? IEEE Signal Processing Magazine 25 (2008), 102-113. DOI 10.1109/MSP.2007.913700
[15] E. Kristiansson: Decreasing Rearrangement and Lorentz $L(p,q)$ Spaces. Master Thesis. Department of Mathematics, Lulea University of Technology, Lulea, 2002; Available at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.111.1244&rep=rep1&type=pdf.
[16] S. Kullback: Information Theory and Statistics. Wiley Publication in Mathematical Statistics, John Wiley & Sons, New York (1959). MR 0103557 | Zbl 0088.10406
[17] E. G. Learned-Miller, J. W. Fisher III: ICA using spacings estimates of entropy. J. Mach. Learn. Res. 4 (2004), 1271-1295. DOI 10.1162/jmlr.2003.4.7-8.1271 | MR 2103630 | Zbl 1061.62007
[18] B. G. Lindsay, W. Yao: Fisher information matrix: A tool for dimension reduction, projection pursuit, independent component analysis, and more. Can. J. Stat. 40 (2012), 712-730. DOI 10.1002/cjs.11166 | MR 2998858 | Zbl 1349.62300
[19] A. W. Marshall, I. Olkin: Inequalities: Theory of Majorization and Its Applications. Mathematics in Science and Engineering 143, Academic Press, New York (1979). DOI 10.1016/B978-0-12-473750-1.50001-1 | MR 0552278 | Zbl 0437.26007
[20] J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen: Deflation-based FastICA with adaptive choices of nonlinearities. IEEE Trans. Signal Process 62 (2014), 5716-5724. DOI 10.1109/TSP.2014.2356442 | MR 3273526 | Zbl 1394.94394
[21] J. Miettinen, K. Nordhausen, H. Oja, S. Taskinen, J. Virta: The squared symmetric FastICA estimator. Signal Process. 131 (2017), 402-411. DOI 10.1016/j.sigpro.2016.08.028
[22] J. Miettinen, S. Taskinen, K. Nordhausen, H. Oja: Fourth moments and independent component analysis. Stat. Sci. 30 (2015), 372-390. DOI 10.1214/15-STS520 | MR 3383886 | Zbl 1332.62196
[23] K. Nordhausen, H. Oja: Independent component analysis: A statistical perspective. WIREs Comput. Stat. 10 (2018), Article ID e1440, 23 pages. DOI 10.1002/wics.1440 | MR 3850449
[24] K. Nordhausen, H. Oja: Robust nonparametric inference. Annu. Rev. Stat. Appl. 5 (2018), 473-500. DOI 10.1146/annurev-statistics-031017-100247 | MR 3774756
[25] H. Oja: On location, scale, skewness and kurtosis of univariate distributions. Scand. J. Stat., Theory Appl. 8 (1981), 154-168. MR 0633040 | Zbl 0525.62020
[26] J. E. Pečarić, F. Proschan, Y. L. Tong: Convex Functions, Partial Orderings, and Statistical Applications. Mathematics in Science and Engineering 187, Academic Press, Boston (1992). DOI 10.1016/S0076-5392(13)60006-5 | MR 1162312 | Zbl 0749.26004
[27] J. V. Ryff: On the representation of doubly stochastic operators. Pac. J. Math. 13 (1963), 1379-1386. DOI 10.2140/pjm.1963.13.1379 | MR 0163171 | Zbl 0125.08405
[28] R. Serfling: Asymptotic relative efficiency in estimation. International Encyclopedia of Statistical Science. Springer, Berlin, 2011, 68-72. DOI 10.1007/978-3-642-04898-2_126
[29] C. E. Shannon: A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948), 379-423. DOI 10.1002/j.1538-7305.1948.tb01338.x | MR 0026286 | Zbl 1154.94303
[30] R. G. Staudte: The shapes of things to come: Probability density quantiles. Statistics 51 (2017), 782-800. DOI 10.1080/02331888.2016.1277225 | MR 3669289 | Zbl 1387.62045
[31] R. G. Staudte, A. Xia: Divergence from, and convergence to, uniformity of probability density quantiles. Entropy 20 (2018), Article ID 317, 10 pages. DOI 10.3390/e20050317 | MR 3862507
[32] W. R. van Zwet: Convex Transformations of Random Variables. Mathematical Centre Tracts 7, Mathematisch Centrum, Amsterdam (1964). MR 0176511 | Zbl 0125.37102
[33] V. Vigneron, C. Jutten: Fisher information in source separation problems. International Conference on Independent Component Analysis and Signal Separation. Lecture Notes in Computer Science 3195, Springer, Berlin, 2004, 168-176. DOI 10.1007/978-3-540-30110-3_22
[34] J. Virta: On characterizations of the covariance matrix. Available at https://arxiv.org/abs/1810.01147 (2018), 11 pages.
[35] J. Virta, K. Nordhausen: On the optimal non-linearities for Gaussian mixtures in FastICA. Latent Variable Analysis and Signal Separation. Theoretical Computer Science and General Issues, Springer, Berlin, 2017, 427-437. DOI 10.1007/978-3-319-53547-0_40
Affiliations: Una Radojičić (corresponding author), Klaus Nordhausen, CSTAT-Computational Statistics, Institute of Statistics & Mathematical Methods in Economics, Vienna University of Technology, Wiedner Hauptstrasse 7, 1040 Vienna, Austria, e-mail: una.radojicic@tuwien.ac.at, klaus.nordhausen@tuwien.ac.at; Hannu Oja, Department of Mathematics and Statistics, University of Turku, 20014 Turku, Finland, e-mail: hannu.oja@utu.fi
