Skip to main content
SpringerLink
Log in

IVDAS: an interactive visual design and analysis system for image data symmetry detection of CNN models

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Convolutional neural networks (CNNs) have achieved breakthrough performance in image recognition and classification. However, it is still a challenge for CNNs to classify images where the classification results strongly depend on the recognition of globally correlated features, for example, the identification of biomolecular structure symmetry in cryo-electron microscopy images. In this work, we improved the traditional CNNs model to solve the problem of cryo-electron microscopy image symmetry recognition and developed a visualization system to help biological researchers to design the best model. Our system consists of the model overview view, the model structure view, and the case view. The system supports a comprehensive analysis of the model’s recognition accuracy, the number of calculations, and the convolutional layers between different models. It also supports structural analysis of models and visual analysis of specific input images. Through the real cases from the biological researchers, we verify that our system can effectively build a model to identify symmetry.

Graphic abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Bairagi VK (2015) Symmetry-based biomedical image compression. J Digit Imaging 28(6):718

    Article  Google Scholar 

  • Celik T (2014) Spatial entropy-based global and local image contrast enhancement. IEEE Trans Image Process 23(12):5298

    Article  MathSciNet  Google Scholar 

  • Chen X, Guan Q, Lo LT, Su S, Ren Z, Ahrens JP, Estrada T (2018) In situ TensorView: in situ visualization of convolutional neural networks. In: 2018 IEEE international conference on big data (big data). IEEE, pp 1899–1904

  • Dosovitskiy A, Brox T (2016) Inverting visual representations with convolutional networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4829–4837. https://academic.microsoft.com/paper/2963989815

  • Erhan D, Bengio Y, Courville A, Vincent P (2009) Visualizing higher-layer features of a deep network. Univ Montreal 1341(3):1

    Google Scholar 

  • He K, Zhang X, Ren S, Jian S (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR)

  • Jarrett K, Kavukcuoglu K, Ranzato M, LeCun Y (2009) What is the best multi-stage architecture for object recognition?. In: 2009 IEEE 12th international conference on computer vision, pp 2146–2153

  • Ji Y, Dan CM, Wei Z, Baker TS (2003) Orientation refinement of virus structures with unknown symmetry. In: 17th international parallel and distributed processing symposium (IPDPS 2003), 22–26 April 2003, Nice, France, CD-ROM/Abstracts Proceedings

  • Jing H, Qinqi W, Shinie C, Fan D, Hongtao G, Fan L (2015) Symmetry image symmetric axis detection positioning method. In: Symmetry image symmetric axis detection positioning method. https://academic.microsoft.com/paper/2839302056

  • Junpeng W, Liang G, Han-Wei S, Hao Y (2018) DQNViz: a visual analytics approach to understand deep Q-networks. IEEE Trans Vis Comput Graph, pp 205-220. https://doi.org/10.1109/TVCG.2018.2864504

  • Kinjo AR, Suzuki H, Yamashita R, Ikegawa Y, Kudou T, Igarashi R, Kengaku Y, Cho H, Standley DM, Nakagawa A, Nakamura H (2012) Protein data bank Japan (PDBj): maintaining a structural data archive and resource description framework format”. Nucleic Acids Res 40:453

    Article  Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, vol 25, no 2

  • Laskowski RA, MacArthur MW, Moss DS, Thornton JM (1993) PROCHECK: a program to check the stereochemical quality of protein structures. J Appl Crystallogr 26(2):283

    Article  Google Scholar 

  • Lin M, Chen Q, Yan S (2014) Network in network. In: ICLR 2014: international conference on learning representations (ICLR) 2014

  • Liu M, Shi J, Li Z, Li C, Zhu J, Liu S (2016) Towards better analysis of deep convolutional neural networks. IEEE Trans Vis Comput Graph 23(1):91

    Article  Google Scholar 

  • Mahendran A, Vedaldi A (2015) Understanding deep image representations by inverting them. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 5188–5196. https://academic.microsoft.com/paper/1915485278

  • Rajasegaran J, Jayasundara V, Jayasekara S, Jayasekara H, Seneviratne S, Rodrigo R (2019) DeepCaps: going deeper with capsule networks. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 10,725–10,733

  • Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in neural information processing systems. pp 3856–3866. https://academic.microsoft.com/paper/2963703618

  • Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR 2015: international conference on learning representations 2015. https://academic.microsoft.com/paper/2962835968

  • Strobelt H, Gehrmann S, Behrisch M, Perer A, Pfister H, Rush AM (2018) SEQ2SEQ-VIS: a visual debugging tool for sequence-to-sequence models. IEEE Trans Vis Comput Graph, pp 1–1 https://doi.org/10.1109/TVCG.2018.2865044

  • Studholme C, Hill DLG, Hawkes DJ (1999) An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognit 32(1):71

    Article  Google Scholar 

  • Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2016) Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence, pp 4278–4284. https://academic.microsoft.com/paper/2964350391

  • Tagari M, Newman R, Chagoyen M, Carazo JM, Henrick K (2002) New electron microscopy database and deposition system. Trends Biochem Sci 27(11):589

    Article  Google Scholar 

  • Velankar S, Best C, Beuth B, Boutselakis CH, Cobley N, Silva AWSD, Dimitropoulos D, Golovin A, Hirshberg M, John M, Krissinel EB, Newman R, Oldfield T, Pajon A, Penkett CJ, Pineda-Castillo J, Sahni G, Sen S, Slowley R, Suarez-Uruena A, Swaminathan J, van Ginkel G, Vranken WF, Henrick K, Kleywegt GJ (2010) PDBe: protein data bank in Europe. Nucleic Acids Res 40:285

    Google Scholar 

  • Wardman J (2012) A symmetry approach to virus architecture. https://academic.microsoft.com/paper/1608112681

  • Xie S, Girshick R, Dollar P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 5987–5995. https://academic.microsoft.com/paper/2549139847

  • Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding neural networks through deep visualization. arXiv:1506.06579

  • Zeiler MD, Krishnan D, Taylor GW, Fergus R (2010) Deconvolutional networks. In: 2010 IEEE computer society conference on computer vision and pattern recognition. https://academic.microsoft.com/paper/2293078015

  • Zeiler MD, Fergus R (2014) ‘Visualizing and understanding convolutional networks. In: 13th European conference on computer vision, ECCV 2014, pp 818–833

  • Zhou ZH, Zhang H, Jakana J, Lu XY, Zhang JQ (2003) Cytoplasmic Polyhedrosis Virus Structure at 8 Ã by electron cryomicroscopy: structural basis of capsid stability and mrna processing regulation. Structure 11(6):651

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Key Research Program of Frontier Sciences, CAS, Grant No. QYZDB-SSW-SMC004-02.

Author information

Authors and Affiliations

  1. Computer Network Information Center, Chinese Academy of Sciences, Beijing, China

    Xiaoyang Han, Guan Li, Xuyi Lu, Guihua Shan & Yangang Wang

  2. University of Chinese Academy of Sciences, Beijing, China

    Xiaoyang Han, Guan Li, Xuyi Lu, Guihua Shan & Yangang Wang

  3. The Ohio State University, Columbus, OH, USA

    Han-Wei Shen

Authors
  1. Xiaoyang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Han-Wei Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuyi Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guihua Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yangang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guihua Shan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, X., Shen, HW., Li, G. et al. IVDAS: an interactive visual design and analysis system for image data symmetry detection of CNN models. J Vis 24, 615–629 (2021). https://doi.org/10.1007/s12650-020-00721-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-020-00721-3

Keywords

Search

Navigation