Virus identification in electron microscopy images by residual mixed attention network

https://doi.org/10.1016/j.cmpb.2020.105766Get rights and content

Highlights

  • We are the first one to incorporate channel attention mechanism with bottom-up and top-down attention to realize the virus identification, and the CAM visualization results illustrate that our method learns well to obtain the attention region of the target virus.

  • On the TEM virus dataset, the top-1 error rate of the proposed method on 12 virus classes is 4.285%, which surpasses that of state-of-the-art networks and even human experts.

  • The fully automated method contributes to the development of medical virology by providing virologists with a high-accuracy approach to recognize viruses and assist in the diagnosis of viruses.

Abstract

Background and Objective: Virus identification in electron microscopy (EM) images is considered as one of the front-line method in pathogen diagnosis and re-emerging infectious agents. However, the existing methods either focused on the detection of a single virus or required large amounts of manual labeling work to segment virus. In this work, we focus on the task of virus classification and propose an effective and simple method to identify different viruses.

Methods: We put forward a residual mixed attention network (RMAN) for virus classification. The proposed network uses channel attention, bottom-up and top-down attention, and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with EM virus images and reducing the burden of manual annotation.

Results: We validate the proposed network through extensive experiments on a transmission electron microscopy virus image dataset. The top-1 error rate of our RMAN on 12 virus classes is 4.285%, which surpasses that of state-of-the-art networks and even human experts. In addition, the ablation study and the visualization of class activation mapping (CAM) further demonstrate the effectiveness of our method.

Conclusions: The proposed automated method contributes to the development of medical virology, which provides virologists with a high-accuracy approach to recognize viruses as well as assist in the diagnosis of viruses.

Introduction

In the early 1930s, Max Knoll and Ernst Ruska constructed the first transmission electron microscope (TEM) [1]. After that, Scientists quickly recognized the potential of this newly developed instrument for revealing the ultrastructures of viruses and other pathogens. Soon after its development, TEM was used in the visualization of viruses such as Poxvirus, Herpesvirus, and Tobacco mosaic virus [2], [3]. The introduction of a negative staining technique that mainly adapts to liquid samples was found by Brenner and Horne in the late 1950s, and this technique uses solutions of heavy metal salts as stains (such as phosphotungstic acid or uranyl acetate) to greatly simplify TEM specimen preparation [4]. Consequently, it became easier to obtain high resolution to reveal far greater details in the ultrastructures of viruses and bacteria and led to the increasing discovery of new pathogens by TEM. Subsequently, more viruses were discovered by TEM, including many of medical importance, such as Norwalk virus, Hepatitis B virus, Hepatitis A virus, Parvovirus B19, Rotavirus, Ebola virus, Henipavirus (Hendra and Nipah), and SARS coronavirus [5], [6], [7], [8], [9], [10], [11], [12], [13]. With the application of TEM for investigating the fine structure of viruses, the differences observed in morphology have long been used as the criteria for virus identification [14]. However, the use of diagnostic TEM appears to have gradually fallen out of favour following the rapid development of modern techniques such as ELISA, PCR and NGS. However, TEM is still considered a catch-all method, and it is very quick and simple compared to most other techniques [15]. TEM still plays a key role in pathogen diagnosis, especially in emergent situations or re-emerging infectious agents [16].

The sizes of most viruses vary from  ~ 20 nm to  ~ 200 nm, which are too small to be observed without TEM. TEM has been a powerful tool in the morphological characterization of viruses because of its high resolution, which can discern virus fine structures such as size, shape, the appearance of the capsid. Generally, the features of viral morphology within a given family are almost identical [17]. Depending on the morphological traits, a virus can be identified to the family level. To date, almost all virus morphological diagnoses are conducted by humans, mainly skilled specialists, which may restrict the virus morphological diagnosis according to human resources. A typical virus diagnostic TEM analysis involves distinguishing the virus particle images from the tremendous amount of information of non-target object images. The manual analysis of TEM images is laborious, exhausting and requires substantial knowledge about viral morphology. Therefore, an automated and accurate virus identification method (classification, detection or segmentation) based on artificial intelligence (AI) is essential for analyzing a large number of virus TEM images.

However, owing to the variety of structures and sizes of different viruses, as well as the background noise and presence of artefacts, automated virus identification in TEM images has proven to be a challenging task. Over the years, several attempts have been developed for virus morphological detection, classification and segmentation in TEM images. Sintorn et al. proposed a refined circular template matching method for the identification of human cytomegalovirus particles [18]. Ong et al. put forward an identification method based on bispectral features that obtains contour and texture information to identify gastroenteric virus [19]. Kylberg et al. employed radial mean intensities to detect virus particles with different shapes [20]. Wen et al. proposed a virus classification method by extracting the virus features through multi-scale principal component analysis [21]. However, the above-mentioned machine learning methods are traditional algorithms. All of them build classifiers by hand-crafted features, which are complicated and yield several false positives. In addition, some viruses are morphologically similar in vision and easy to confuse, such as Adeno-associated virus (AAV) and Human enterovirus 71 (HEV71). As shown in Fig. 1, the differences between these viruses are unobvious, which increases the difficulty of classification and recognition.

Currently, the AI technique has been widely used to solve the classification and segmentation challenges in medical imaging due to its extraordinary performance [22], [23], and it is possible to combine AI with virus recognition and classification to liberate specialists from laborious work. In the field of AI research, the fine-grained image classification is a different task since the shape and geometric characteristics of fine-grained categories being very similar; therefore, it is important to identify fine-grained images through the differences between key parts. This insight was also applied to virus classification. Since the attention mechanism is effectively used in fine-grained categorization [24], [25], we put forward a mixed attention network to enhance the ability of the deep neural network to identify viruses.

In this study, we proposed a residual mixed attention network (RMAN) for virus classification. The realistic application scenarios of our method is as follows: Firstly, the suspected virus samples are processed by negative staining and imaged through TEM. In the process of imaging, the staffs select areas where they are interested or objects may be viruses for shooting. Next, the proposed network is used to predict the type of virus in the TEM images. In closing, virologist makes the final judgment according to the prediction results. The proposed network applied an effective channel attention, bottom-up and top-down attention model with a valid residual network. We evaluated our network on a TEM virus dataset and compared our results with other state-of-the-art classification networks [26], [27], [28], [29], [30], [31], [32] and human experts. The experimental results demonstrated that our network outperformed state-of-the-art networks and even human experts, which could provide virologists with a very high-accuracy approach for identifying viruses. Additionally, we compared the CAM visualization of our network with other baseline networks, and the results further demonstrated the validity of our method.

Section snippets

TEM specimen preparation

In this work, we selected 12 viruses from different families as the experimental dataset, and the details are illustrated in Table 1. Human adenovirus type 5 (HAdv5), Herpes simplex virus-1 (HSV-1), Rotavirus A (RV-A), HEV71, and H1N1 Influenza virus (FLUAV) were cultured in HEK293, Hep2, MA104, RD-A cells and MDCK, respectively. Cultures were inactivated with 2% formaldehyde and concentrated or purified by ultracentrifugation. Formaldehyde-inactivated Marburg virus (MBGV) negative stained

Experiments on virus dataset

Experiments were conducted to evaluate the performance on the dataset of TEM images. The proposed network was compared with other promising AI methods: SqueezeNet [26], Vgg19 [27], ResNet50 [28], SE-integrated ResNet50 (SE-ResNet50) [32], GoogLeNet [29], DenseNet101 [30], and residual attention network (RAN92) [31]. Thereinto, Vgg19, ResNet50, GoogLeNet and DenseNet101 are outstanding networks that have achieved significant performance in the challenge of image classification [34], and

Discussion

Currently, several AI techniques have achieved good performance in the tasks of virus detection and segmentation. Devan et al. employed a transfer learning approach to detect HCMV nucleocapsids [44]. Ito et al. also designed a simple convolutional neural network (CNN) to detect feline calicivirus [45]. Both methods achieved promising results in virus detection and recognition. However, the above methods mainly focused on the detection of a single virus, and they have not been shown to be

Funding Statement

This research was supported by National Natural Science Foundation of China (No. 61673381, No. 61871177, No. 31472001), Special Program of Beijing Municipal Science and Technology Commission (No. Z161100000216146), Scientific Research Instrument and Equipment Development Project of the CAS (No. YZ201671) and Strategic Priority Research Program of the CAS (No. XDB02060001), Science Foundation for the State Key Laboratory for Infectious Disease Prevention and Control of China (No. 2014SKLID206),

Declaration of Competing Interest

The authors declared that they have no competing interests.

References (46)

  • H. Ong et al.

    Identification of gastroenteric viruses by electron microscopy using higher order spectral features

    J. Clin. Virol.

    (2005)
  • Z. Wen et al.

    Virus image classification using multi-scale completed local binary pattern features extracted from filtered images by multi-scale principal component analysis

    Pattern Recognit. Lett.

    (2016)
  • G. Litjens et al.

    A survey on deep learning in medical image analysis

    Med. Image Anal.

    (2017)
  • T. Kooi et al.

    Large scale deep learning for computer aided detection of mammographic lesions

    Med. Image Anal.

    (2017)
  • Z. Wu et al.

    Wider or deeper: revisiting the resnet model for visual recognition

    Pattern Recognit.

    (2019)
  • D.J. Matuszewski et al.

    Reducing the U-Net size for practical scenarios: virus recognition in electron microscopy images

    Comput. Methods Programs Biomed.

    (2019)
  • M. Knoll et al.

    Das Elektronenmikroskop

    Z. Phys.

    (1932)
  • F. Nagler et al.

    The use of the electron microscope in diagnosis of variola, vaccinia, and varicella

    J. Bacteriol.

    (1948)
  • A.Z. Kapikian et al.

    Visualization by immune electron microscopy of a 27-nm particle associated with acute infectious nonbacterial gastroenteritis

    J. Virol.

    (1972)
  • S.M. Feinstone et al.

    Hepatitis a: detection by immune electron microscopy of a viruslike antigen associated with acute illness

    Science

    (1973)
  • T.G. Ksiazek et al.

    A novel coronavirus associated with severe acute respiratory syndrome

    N. Engl. J. Med.

    (2003)
  • M. Nermut et al.

    Animal Virus Structure

    (1987)
  • P.R. Hazelton et al.

    Electron microscopy for rapid diagnosis of emerging infectious agents

    Emerg. Infect. Dis.

    (2003)
  • Cited by (12)

    • Recent insights into aquatic viruses: Emerging and reemerging pathogens, molecular features, biological effects, and novel investigative approaches

      2022, Water Biology and Security
      Citation Excerpt :

      A residual mixed attention network (RMAN) for virus classification from electron microscopy (EM) images has been reported. The proposed network uses channel attention and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with electron microscopy virus images, identifying virus species, and reducing the burden of manual annotation (Xiao et al., 2021). Detection of viruses in water is complicated and different research methods have different practical application.

    • TEM virus images: Benchmark dataset and deep learning classification

      2021, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      In the field of automatic virus recognition in transmission electron microscopy (TEM) images, there is only a handful of published papers. Many of which come from our group [12,13,14,15,16] and some from others [19,20,21,22]. This low number of publications is likely due to the shortage of publicly available image datasets suitable for machine learning.

    • Deep Learning for Imaging and Detection of Microorganisms

      2021, Trends in Microbiology
      Citation Excerpt :

      In addition to single virus detection, deep learning has been used to classify TEM images of a dozen virus classes with variable structure and size, including morphologically similar viruses, such as adeno-associated virus (AAV) and human enterovirus 71 (EV71). The proposed method achieved a better result than human experts [15]. A TEM image dataset of 15 virus classes [16] (Table 1) is publicly available for training deep-learning models for virus detection and segmentation [17].

    View all citing articles on Scopus
    View full text