Skip to main content
SpringerLink
Log in

Predicted Robustness as QoS for Deep Neural Network Models

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

The adoption of deep neural network (DNN) model as the integral part of real-world software systems necessitates explicit consideration of their quality-of-service (QoS). It is well-known that DNN models are prone to adversarial attacks, and thus it is vitally important to be aware of how robust a model’s prediction is for a given input instance. A fragile prediction, even with high confidence, is not trustworthy in light of the possibility of adversarial attacks. We propose that DNN models should produce a robustness value as an additional QoS indicator, along with the confidence value, for each prediction they make. Existing approaches for robustness computation are based on adversarial searching, which are usually too expensive to be excised in real time. In this paper, we propose to predict, rather than to compute, the robustness measure for each input instance. Specifically, our approach inspects the output of the neurons of the target model and trains another DNN model to predict the robustness. We focus on convolutional neural network (CNN) models in the current research. Experiments show that our approach is accurate, with only 10%–34% additional errors compared with the offline heavy-weight robustness analysis. It also significantly outperforms some alternative methods. We further validate the effectiveness of the approach when it is applied to detect adversarial attacks and out-of-distribution input. Our approach demonstrates a better performance than, or at least is comparable to, the state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Andor D, Alberti C, Weiss D, Severyn A, Presta A, Ganchev K, Petrov S, Collins M. Globally normalized transition-based neural networks. arXiv:1603.06042, 2016. https://arxiv.org/abs/1603.06042, June 2020.

  2. Hinton G, Deng L, Yu D, Dahl G, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Kingsbury B. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012, 29(6): 82-97.

    Article  Google Scholar 

  3. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.

  4. Wang X, Huang C, Yao L, Benatallah B, Dong M. A survey on expert recommendation in community question answering. Journal of Computer Science and Technology, 2018, 33(4): 625-653.

    Article  Google Scholar 

  5. Liu Q, Zhao H K, Wu L, Li Z, Chen E H. Illuminating recommendation by understanding the explicit item relations. Journal of Computer Science and Technology, 2018, 33(4): 739-755.

    Article  Google Scholar 

  6. Silver D, Huang A, Maddison C J et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529(7587): 484-489.

    Article  Google Scholar 

  7. Ameur H, Jamoussi S, Hamadou A B. A new method for sentiment analysis using contextual auto-encoders. Journal of Computer Science and Technology, 2018, 33(6): 1307-1319.

    Article  Google Scholar 

  8. Bojarski M, Testa D D, Dworakowski D et al. End to end learning for self-driving cars. arXiv:1604.07316, 2016. https://arxiv.org/abs/1604.07316, June 2020.

  9. Esteva A, Kuprel B, Novoa R A, Ko J, Swetter S M, Blau H M, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 2017, 542(7639): 115-118.

    Article  Google Scholar 

  10. Yuan Z, Lu Y, Wang Z, Xue Y. Droid-Sec: Deep learning in Android malware detection. ACM SIGCOMM Computer Communication Review, 2014, 44(4): 371-372.

    Article  Google Scholar 

  11. Li Z, Ma X, Xu C, Xu J, Cao C, Lü J. Operational calibration: Debugging confidence errors for DNNs in the field. arXiv:1910.02352, 2019. https://arxiv.org/abs/1910.02352, Sept. 2020.

  12. Li Z, Ma X, Xu C, Cao C, Xu J, Lü J. Boosting operational DNN testing efficiency through conditioning. In Proc. the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, August 2019, pp.499-509.

  13. LeCun Y, Bengio Y, Hinton G. Deep Learning. MIT Press, 2016.

    Google Scholar 

  14. Burrell J. How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big Data & Society, 2016, 3(1): Article No. 2053951715622512.

  15. Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014. https://arxiv.org/abs/1412.6572, June 2020.

  16. Moosavi-Dezfooli S, Fawzi A, Frossard P. DeepFool: A simple and accurate method to fool deep neural networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.2574-2582.

  17. Carlini N, Wagner D. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proc. the 10th ACM Workshop on Artificial Intelligence and Security, November 2017, pp.3-14.

  18. Athalye A, Carlini N, Wagner D. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv:1802.00420, 2018. https://arxiv.org/abs/1802.00420, June 2020.

  19. Katz G, Barrett C, Dill D L, Julian K, Kochenderfer M J. Reluplex: An efficient SMT solver for verifying deep neural networks. In Proc. the 29th International Conference on Computer Aided Verification, July 2017, pp.97-117.

  20. Bastani O, Ioannou Y, Lampropoulos L, Vytiniotis D, Nori A, Criminisi A. Measuring neural net robustness with constraints. In Proc. the Annual Conference on Neural Information Processing Systems, December 2016, pp.2613-2621.

  21. Hendrycks D, Gimpel K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv:1610.02136, 2016. https://arxiv.org/abs/1610.02136, June 2020.

  22. Weng T, Zhang H, Chen H, Song Z, Hsieh C, Boning D, Dhillon I S, Daniel L. Towards fast computation of certified robustness for ReLU networks. arXiv:1804.09699, 2018. https://arxiv.org/abs/1804.09699, June 2020.

  23. Singh G, Gehr T, Püschel M, Vechev M. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 2019, 3(POPL): Article No. 41.

  24. Carlini N, Wagner D. Towards evaluating the robustness of neural networks. In Proc. the 2017 IEEE Symposium on Security and Privacy, May 2017, pp.39-57.

  25. Feinman R, Curtin R R, Shintre S, Gardner A B. Detecting adversarial samples from artifacts. arXiv:1703.00410, 2017. https://arxiv.org/abs/1703.00410, June 2020.

  26. Ma X, Li B, Wang Y, Erfani S M, Wijewickrema S, Schoenebeck G, Song D, Houle M E, Bailey J. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv:1801.02613, 2018. https://arxiv.org/abs/1801.02613, June 2020.

  27. Wang Y, Li Z, Xu J, Yu P, Ma X. Fast robustness prediction for deep neural network. In Proc. the 11th Asia-Pacific Symposium on Internetware, Oct. 2019.

  28. Kurakin A, Goodfellow I, Bengio S. Adversarial examples in the physical world. arXiv:1607.02533, 2016. https://arxiv.org/abs/1607.02533, June 2020.

  29. Papernot N, McDaniel P, Jha S, Fredrikson M, Celik Z B, Swami A. The limitations of deep learning in adversarial settings. In Proc. the 2016 IEEE European Symposium on Security and Privacy, March 2016, pp.372-387.

  30. Huang X, Kroening D, Kwiatkowska M, Ruan W, Sun Y, Thamo E, Wu M, Yi X. Safety and trustworthiness of deep neural networks: A survey. arXiv:1812.08342, 2018. https://arxiv.org/abs/1812.08342, June 2020.

  31. Huang X, Kwiatkowska M, Wang S, Wu M. Safety verification of deep neural networks. In Proc. the 29th International Conference on Computer Aided Verification, July 2017, pp.3-29.

  32. Wong E, Kolter J Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv:1711.00851, 2017. https://arxiv.org/abs/1711.00851, June 2020.

  33. Gopinath D, Pasareanu C S, Wang K, Zhang M, Khurshid S. Symbolic execution for attribution and attack synthesis in neural networks. In Proc. the 41st IEEE/ACM International Conference on Software Engineering, May 2019, pp.282-283.

  34. Pei K, Cao Y, Yang J, Jana S. DeepXplore: Automated whitebox testing of deep learning systems. In Proc. the 26th Symposium on Operating Systems Principles, October 2017, pp.1-18.

  35. Ma L, Juefei-Xu F, Zhang F et al. DeepGauge: Multigranularity testing criteria for deep learning systems. In Proc. the 33rd ACM/IEEE International Conference on Automated Software Engineering, September 2018, pp.120-131.

  36. Ma L, Zhang F, Xue M, Li B, Liu Y, Zhao J, Wang Y. Combinatorial testing for deep learning systems. arXiv:1806.07723, 2018. https://arxiv.org/abs/1806.07723, June 2020.

  37. Zong B, Song Q, Min M, Cheng W, Lumezanu C, Cho D, Chen H. Deep autoencoding Gaussian mixture model for unsupervised anomaly detection. In Proc. International Conference on Learning Representations, February 2018.

  38. Santhanam G K, Grnarova P. Defending against adversarial attacks by leveraging an entire GAN. arXiv:1805.10652, 2018. https://arxiv.org/abs/1805.10652, June 2020.

  39. Grosse K, Manoharan P, Papernot N, Backes M, McDaniel P. On the (statistical) detection of adversarial examples. arXiv:1702.06280, 2017. https://arxiv.org/abs/1702.06280, June 2020.

  40. Xu W, Evans D, Qi Y. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv:1704.01155, 2017. https://arxiv.org/abs/1704.01155, June 2020.

  41. Benesty J, Chen J, Huang Y, Cohen I. Pearson correlation coefficient. In Noise Reduction in Speech Processing, Cohen I, Huang Y, Chen J, Benesty J (eds.), Springer, 2009, pp.1-4.

  42. LeCun L, Boser B, Denker J S, Henderson D, Howard R E, Hubbard W, Jackel L D. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 1(4): 541-551.

    Article  Google Scholar 

  43. Krizhevsky A. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009. http://www.cs.toronto.edu/_kriz/learning-features-2009-TR.pdf, June 2020.

  44. Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng A Y. Reading digits in natural images with unsupervised feature learning. In Proc. the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Dec. 2011.

  45. Deng J, Dong W, Socher R, Li L J, Li K, Li F F. ImageNet: A large-scale hierarchical image database. In Proc. the 2009 IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp.248-255.

  46. LeCun L, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

    Article  Google Scholar 

  47. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. https://arxiv.org/abs/1409.1556, June 2020.

  48. Kim J, Feldt R, Yoo S. Guiding deep learning system testing using surprise adequacy. In Proc. the 41st International Conference on Software Engineering, May 2019, pp.1039-1049.

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, China

    Yue-Huan Wang, Ze-Nan Li, Jing-Wei Xu, Ping Yu, Taolue Chen & Xiao-Xing Ma

  2. Department of Computer Science, University of Surrey, Guilford, GU2 7XH, UK

    Taolue Chen

Authors
  1. Yue-Huan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ze-Nan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing-Wei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ping Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Taolue Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiao-Xing Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing-Wei Xu.

Electronic supplementary material

ESM 1

(PDF 316 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, YH., Li, ZN., Xu, JW. et al. Predicted Robustness as QoS for Deep Neural Network Models. J. Comput. Sci. Technol. 35, 999–1015 (2020). https://doi.org/10.1007/s11390-020-0482-6

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-0482-6

Keywords

Search

Navigation