Abstract
We describe our submission to the Extreme Value Analysis 2019 Data Challenge in which teams were asked to predict extremes of sea surface temperature anomaly within spatio-temporal regions of missing data. We present a computational framework which reconstructs missing data using convolutional deep neural networks. Conditioned on incomplete data, we employ autoencoder-like models as multivariate conditional distributions from which possible reconstructions of the complete dataset are sampled using imputed noise. In order to mitigate bias introduced by any one particular model, a prediction ensemble is constructed to create the final distribution of extremal values. Our method does not rely on expert knowledge in order to accurately reproduce dynamic features of a complex oceanographic system with minimal assumptions. The obtained results promise reusability and generalization to other domains.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Asadi, R., Regan, A.: A convolution recurrent autoencoder for spatio-temporal missing data imputation. arXiv:1904.12413v1 (2019)
Behnke, S.: Hierarchical Neural Networks for Image Interpretation, Lecture Notes in Computer Science, vol. 2766. Springer (2003)
Cao, W., Wang, D., Li, J., Zhou, H., Li, L., Li, Y.: BRITS: Bidirectional recurrent imputation for time series. Adv. Neural Inf. Process. Syst. 31, pp. 6775–6785 http://papers.nips.cc/paper/7911-brits-bidirectional-recurrent-imputation-for-time-series (2018)
Che, Z., Purushotham, S., Cho, K., Sontag, D., Liu, Y.: Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8(1):6085–. https://doi.org/10.1038/s41598-018-24271-9 (2018)
Davison, A.C., Padoan, S.A., Ribatet, M.: Statistical modeling of spatial extremes. Statist. Sci. 27(2):161–186. https://doi.org/10.1214/11-STS376 (2012)
Davison, A., Huser, R.: Statistics of extremes. Ann. Rev. Stat. Appl. 2 (1), 203–235 (2015). https://doi.org/10.1146/annurev-statistics-010814-020133
Davison, A.C., Huser, R., Thibaud, E.: Spatial Extremes. In: Gelfand, AE, Fuentes, M, Smith, R L (eds.) Handbook of Environmental and Ecological Statistics. https://doi.org/10.1201/9781315152509, pp 711–744. CRC Press (2019)
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv:1406.2661 (2014)
Goodfellow, I., Bengio, Y, Courville, A.: Deep Learning. Adaptive computation and machine learning. MIT Press, http://www.deeplearningbook.org (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv:1512.03385 (2015)
Howard, J., et al.: fastai. https://github.com/fastai/fastai (2018)
Huser, R.: Editorial: EVA 2019 data competition on spatio-temporal prediction of Red Sea surface temperature extremes. Extremes, https://doi.org/10.1007/s10687-019-00369-9 (2020)
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. 1502.03167 (2015)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. 1412.6980v9 (2017)
Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Advances in neural information processing systems, pp. 971–980 (2017)
Kramer, M.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J 37, 233–243 (1991)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25:1106–1114. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (2012)
Le, T., Nguyen, T.D., Phung, D.: Kgan: How to break the minimax game in gan. arXiv:171101744 (2017)
Lee, J.W., Park, S.C.: Artificial neural network-based data recovery system for the time series of tide stations. J. Coast. Res. 32(1):213–224. https://doi.org/10.2112/JCOASTRES-D-14-00233.1, https://meridian.allenpress.com/jcr/article-pdf/32/1/213/1648085/jcoastres-d-14-00233_1.pdf (2015)
Li, S.C., Jiang, B, Marlin, B.M.: MisGAN: Learning from incomplete data with generative adversarial networks. CoRR arXiv:1902.09599 (2019)
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J, Han, J.: On the variance of the adaptive learning rate and beyond. arXiv:http://arxiv.org/pdf/1908.03265v3 (2020)
Nitish S, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R: Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(56):1929–1958, http://jmlr.org/papers/v15/srivastava14a.html (2014)
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: PyTorch: An imperative style, high-performance deep learning library. In: Wallach, H.M, Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019. http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library, pp 8024–8035. NeurIPS 2019, Vancouver (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. https://doi.org/10.1007/978-3-319-24574-4_28, pp 234–241. Springer International Publishing, Cham (2015)
Rossiev, A., Makarenko, N., Kuandykov, Y., Dergachev, V.: Recovering data gaps through neural network methods. Int. J. Geomagn. Aeron. 3, 191–197 (2002)
Schlemper, J., Caballero, J., Hajnal, J.V., Price, A., Rueckert, D.: A deep cascade of convolutional neural networks for MR image reconstruction. arXiv:http://arxiv.org/pdf/1703.00555v1 (2017)
Smith, L.N.: A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay. CoRR arXiv:1803.09820 (2018)
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. https://papers.nips.cc/paper/7181-attention-is-all-you-need, vol. 30, pp 5998–6008. Curran Associates, Inc. (2017)
Wei, X., Gong, B., Liu, Z., Lu, W., Wang, L.: Improving the improved training of Wasserstein GANs: A consistency term and its dual effect. arXiv:http://arxiv.org/pdf/1803.01541v1 (2018)
Wright, L., et al.: Ranger. https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer (2019)
Zhang, W.: Shift-Invariant Pattern Recognition Neural Network and Its Optical Architecture. In: Proceedings of Annual Conference of the Japan Society of Applied Physics (1988)
Zhang, M.R., Lucas, J., Hinton, G., Ba, J.: Lookahead optimizer: k steps forward, 1 step back. arXiv:http://arxiv.org/pdf/1907.08610v2 (2019)
Acknowledgements
We thank Ivan Balog for enlightening discussions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ivek, T., Vlah, D. BlackBox: Generalizable reconstruction of extremal values from incomplete spatio-temporal data. Extremes 24, 145–162 (2021). https://doi.org/10.1007/s10687-020-00396-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10687-020-00396-x
Keywords
- Convolutional neural network
- Data reconstruction
- Deep learning
- Extreme Value Analysis Conference challenge
- Ensemble
- Spatio-temporal extremes