Virus identification in electron microscopy images by residual mixed attention network
Introduction
In the early 1930s, Max Knoll and Ernst Ruska constructed the first transmission electron microscope (TEM) [1]. After that, Scientists quickly recognized the potential of this newly developed instrument for revealing the ultrastructures of viruses and other pathogens. Soon after its development, TEM was used in the visualization of viruses such as Poxvirus, Herpesvirus, and Tobacco mosaic virus [2], [3]. The introduction of a negative staining technique that mainly adapts to liquid samples was found by Brenner and Horne in the late 1950s, and this technique uses solutions of heavy metal salts as stains (such as phosphotungstic acid or uranyl acetate) to greatly simplify TEM specimen preparation [4]. Consequently, it became easier to obtain high resolution to reveal far greater details in the ultrastructures of viruses and bacteria and led to the increasing discovery of new pathogens by TEM. Subsequently, more viruses were discovered by TEM, including many of medical importance, such as Norwalk virus, Hepatitis B virus, Hepatitis A virus, Parvovirus B19, Rotavirus, Ebola virus, Henipavirus (Hendra and Nipah), and SARS coronavirus [5], [6], [7], [8], [9], [10], [11], [12], [13]. With the application of TEM for investigating the fine structure of viruses, the differences observed in morphology have long been used as the criteria for virus identification [14]. However, the use of diagnostic TEM appears to have gradually fallen out of favour following the rapid development of modern techniques such as ELISA, PCR and NGS. However, TEM is still considered a catch-all method, and it is very quick and simple compared to most other techniques [15]. TEM still plays a key role in pathogen diagnosis, especially in emergent situations or re-emerging infectious agents [16].
The sizes of most viruses vary from ~ 20 nm to ~ 200 nm, which are too small to be observed without TEM. TEM has been a powerful tool in the morphological characterization of viruses because of its high resolution, which can discern virus fine structures such as size, shape, the appearance of the capsid. Generally, the features of viral morphology within a given family are almost identical [17]. Depending on the morphological traits, a virus can be identified to the family level. To date, almost all virus morphological diagnoses are conducted by humans, mainly skilled specialists, which may restrict the virus morphological diagnosis according to human resources. A typical virus diagnostic TEM analysis involves distinguishing the virus particle images from the tremendous amount of information of non-target object images. The manual analysis of TEM images is laborious, exhausting and requires substantial knowledge about viral morphology. Therefore, an automated and accurate virus identification method (classification, detection or segmentation) based on artificial intelligence (AI) is essential for analyzing a large number of virus TEM images.
However, owing to the variety of structures and sizes of different viruses, as well as the background noise and presence of artefacts, automated virus identification in TEM images has proven to be a challenging task. Over the years, several attempts have been developed for virus morphological detection, classification and segmentation in TEM images. Sintorn et al. proposed a refined circular template matching method for the identification of human cytomegalovirus particles [18]. Ong et al. put forward an identification method based on bispectral features that obtains contour and texture information to identify gastroenteric virus [19]. Kylberg et al. employed radial mean intensities to detect virus particles with different shapes [20]. Wen et al. proposed a virus classification method by extracting the virus features through multi-scale principal component analysis [21]. However, the above-mentioned machine learning methods are traditional algorithms. All of them build classifiers by hand-crafted features, which are complicated and yield several false positives. In addition, some viruses are morphologically similar in vision and easy to confuse, such as Adeno-associated virus (AAV) and Human enterovirus 71 (HEV71). As shown in Fig. 1, the differences between these viruses are unobvious, which increases the difficulty of classification and recognition.
Currently, the AI technique has been widely used to solve the classification and segmentation challenges in medical imaging due to its extraordinary performance [22], [23], and it is possible to combine AI with virus recognition and classification to liberate specialists from laborious work. In the field of AI research, the fine-grained image classification is a different task since the shape and geometric characteristics of fine-grained categories being very similar; therefore, it is important to identify fine-grained images through the differences between key parts. This insight was also applied to virus classification. Since the attention mechanism is effectively used in fine-grained categorization [24], [25], we put forward a mixed attention network to enhance the ability of the deep neural network to identify viruses.
In this study, we proposed a residual mixed attention network (RMAN) for virus classification. The realistic application scenarios of our method is as follows: Firstly, the suspected virus samples are processed by negative staining and imaged through TEM. In the process of imaging, the staffs select areas where they are interested or objects may be viruses for shooting. Next, the proposed network is used to predict the type of virus in the TEM images. In closing, virologist makes the final judgment according to the prediction results. The proposed network applied an effective channel attention, bottom-up and top-down attention model with a valid residual network. We evaluated our network on a TEM virus dataset and compared our results with other state-of-the-art classification networks [26], [27], [28], [29], [30], [31], [32] and human experts. The experimental results demonstrated that our network outperformed state-of-the-art networks and even human experts, which could provide virologists with a very high-accuracy approach for identifying viruses. Additionally, we compared the CAM visualization of our network with other baseline networks, and the results further demonstrated the validity of our method.
Section snippets
TEM specimen preparation
In this work, we selected 12 viruses from different families as the experimental dataset, and the details are illustrated in Table 1. Human adenovirus type 5 (HAdv5), Herpes simplex virus-1 (HSV-1), Rotavirus A (RV-A), HEV71, and H1N1 Influenza virus (FLUAV) were cultured in HEK293, Hep2, MA104, RD-A cells and MDCK, respectively. Cultures were inactivated with 2% formaldehyde and concentrated or purified by ultracentrifugation. Formaldehyde-inactivated Marburg virus (MBGV) negative stained
Experiments on virus dataset
Experiments were conducted to evaluate the performance on the dataset of TEM images. The proposed network was compared with other promising AI methods: SqueezeNet [26], Vgg19 [27], ResNet50 [28], SE-integrated ResNet50 (SE-ResNet50) [32], GoogLeNet [29], DenseNet101 [30], and residual attention network (RAN92) [31]. Thereinto, Vgg19, ResNet50, GoogLeNet and DenseNet101 are outstanding networks that have achieved significant performance in the challenge of image classification [34], and
Discussion
Currently, several AI techniques have achieved good performance in the tasks of virus detection and segmentation. Devan et al. employed a transfer learning approach to detect HCMV nucleocapsids [44]. Ito et al. also designed a simple convolutional neural network (CNN) to detect feline calicivirus [45]. Both methods achieved promising results in virus detection and recognition. However, the above methods mainly focused on the detection of a single virus, and they have not been shown to be
Funding Statement
This research was supported by National Natural Science Foundation of China (No. 61673381, No. 61871177, No. 31472001), Special Program of Beijing Municipal Science and Technology Commission (No. Z161100000216146), Scientific Research Instrument and Equipment Development Project of the CAS (No. YZ201671) and Strategic Priority Research Program of the CAS (No. XDB02060001), Science Foundation for the State Key Laboratory for Infectious Disease Prevention and Control of China (No. 2014SKLID206),
Declaration of Competing Interest
The authors declared that they have no competing interests.
References (46)
- et al.
Helmut Ruska and the visualisation of viruses
Lancet
(2000) - et al.
A negative staining method for high resolution electron microscopy of viruses
Biochim. Biophys. Acta
(1959) - et al.
Virus-like particles in serum of patients with Australia-antigen-associated hepatitis
Lancet
(1970) - et al.
Parvovirus-like particles in human sera
Lancet
(1975) - et al.
Rotavirus-like agent in adult non-bacterial diarrhoea in China
Lancet
(1983) - et al.
Isolation and partial characterisation of a new virus causing acute hemorrhagic fever in Zaire
Lancet
(1977) - et al.
Ultrastructure of equine morbillivirus
Virus Res.
(1996) - et al.
Ultrastructure of Hendra virus and Nipah virus within cultured cells and host animals
Microbes Infect.
(2001) - et al.
Application of transmission electron microscopy to the clinical study of viral and bacterial infections: present and future
Micron
(2006) - et al.
A refined circular template matching method for classification of human cytomegalovirus capsids in TEM images
Comput. Methods Programs Biomed.
(2004)
Identification of gastroenteric viruses by electron microscopy using higher order spectral features
J. Clin. Virol.
Virus image classification using multi-scale completed local binary pattern features extracted from filtered images by multi-scale principal component analysis
Pattern Recognit. Lett.
A survey on deep learning in medical image analysis
Med. Image Anal.
Large scale deep learning for computer aided detection of mammographic lesions
Med. Image Anal.
Wider or deeper: revisiting the resnet model for visual recognition
Pattern Recognit.
Reducing the U-Net size for practical scenarios: virus recognition in electron microscopy images
Comput. Methods Programs Biomed.
Das Elektronenmikroskop
Z. Phys.
The use of the electron microscope in diagnosis of variola, vaccinia, and varicella
J. Bacteriol.
Visualization by immune electron microscopy of a 27-nm particle associated with acute infectious nonbacterial gastroenteritis
J. Virol.
Hepatitis a: detection by immune electron microscopy of a viruslike antigen associated with acute illness
Science
A novel coronavirus associated with severe acute respiratory syndrome
N. Engl. J. Med.
Animal Virus Structure
Electron microscopy for rapid diagnosis of emerging infectious agents
Emerg. Infect. Dis.
Cited by (12)
Machine learning for cross-scale microscopy of viruses
2023, Cell Reports MethodsRecent insights into aquatic viruses: Emerging and reemerging pathogens, molecular features, biological effects, and novel investigative approaches
2022, Water Biology and SecurityCitation Excerpt :A residual mixed attention network (RMAN) for virus classification from electron microscopy (EM) images has been reported. The proposed network uses channel attention and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with electron microscopy virus images, identifying virus species, and reducing the burden of manual annotation (Xiao et al., 2021). Detection of viruses in water is complicated and different research methods have different practical application.
TEM virus images: Benchmark dataset and deep learning classification
2021, Computer Methods and Programs in BiomedicineCitation Excerpt :In the field of automatic virus recognition in transmission electron microscopy (TEM) images, there is only a handful of published papers. Many of which come from our group [12,13,14,15,16] and some from others [19,20,21,22]. This low number of publications is likely due to the shortage of publicly available image datasets suitable for machine learning.
Deep Learning for Imaging and Detection of Microorganisms
2021, Trends in MicrobiologyCitation Excerpt :In addition to single virus detection, deep learning has been used to classify TEM images of a dozen virus classes with variable structure and size, including morphologically similar viruses, such as adeno-associated virus (AAV) and human enterovirus 71 (EV71). The proposed method achieved a better result than human experts [15]. A TEM image dataset of 15 virus classes [16] (Table 1) is publicly available for training deep-learning models for virus detection and segmentation [17].
VISN: virus instance segmentation network for TEM images using deep attention transformer
2023, Briefings in Bioinformatics