Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy

A preprint version of the article is available at arXiv.

Abstract

Fluorescence microscopy allows for a detailed inspection of cells, cellular networks and anatomical landmarks by staining with a variety of carefully selected markers visualized as colour channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers and therefore applicable to a very restricted number of experimental settings. We herein propose ‘marker sampling and excite’—a neural network approach with a modality sampling strategy and a novel attention module that together enable (1) flexible training with heterogeneous datasets with combinations of markers and (2) successful utility of learned models on arbitrary subsets of markers prospectively. We show that our single neural network solution performs comparably to an upper bound scenario in which an ensemble of many networks is naively trained for each possible marker combination separately. We also demonstrate the feasibility of this framework in high-throughput biological analysis by revising a recent quantitative characterization of bone-marrow vasculature in three-dimensional confocal microscopy datasets and further confirm the validity of our approach on another substantially different dataset of microvessels in foetal liver tissues. Not only can our work substantially ameliorate the use of deep learning in fluorescence microscopy analysis, but it can also be utilized in other fields with incomplete data acquisitions and missing modalities.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Segmentation of different classes with different UNet models trained with specific marker combinations.
Fig. 2: Segmentation with a single CNN model for all marker combinations.
Fig. 3: Results with attention modules.
Fig. 4: Comparison of proposed CNN models to an upper bound when training with heterogeneous combinations of markers.
Fig. 5: Vasculature segmentation and quantification in bone marrow with MS–ME.
Fig. 6: Study of our proposed models in the segmentation of foetal liver vasculature with different marker combinations.

Similar content being viewed by others

Data availability

The labelled dataset employed for training and evaluation of the models described is included as a single HDF5 file within a CodeOcean capsule in https://codeocean.com/capsule/8424915/tree/v1 (ref. 46).

Code availability

The code employed for training the models described in this paper is publicly available on the CodeOcean platform as https://codeocean.com/capsule/8424915/tree/v1 (ref. 46). This capsule also includes the trained models employed for the different presented figures. MS–ME is also implemented within MiNTiF, our Fiji plugin for ImageJ for user-friendly training and deployment of CNNs by non-experts of deep learning: https://github.com/CAiM-lab/MiNTiF.

References

  1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  2. Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017).

    Article  Google Scholar 

  3. Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 234–241 (Springer, 2015).

  4. Zhang, Y. & Yang, Q. An overview of multi-task learning. Natl Sci. Rev. 5, 30–43 (2018).

    Article  Google Scholar 

  5. Wang, M. & Deng, W. Deep visual domain adaptation: a survey. Neurocomputing 312, 135–153 (2018).

    Article  Google Scholar 

  6. Guo, S.-M. et al. Revealing architectural order with quantitative label-free imaging and deep learning. eLife 9, e55502 (2020).

    Article  Google Scholar 

  7. Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).

    Article  Google Scholar 

  8. Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173, 792–803 (2018).

    Article  Google Scholar 

  9. Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

    Article  Google Scholar 

  10. Li, R. et al. Deep learning based imaging data completion for improved brain disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention 305–312 (Springer, 2014).

  11. Iglesias, J. E. et al. Is synthesizing MRI contrast useful for inter-modality analysis? In International Conference on Medical Image Computing and Computer-Assisted Intervention 631–638 (Springer, 2013).

  12. Chartsias, A., Joyce, T., Giuffrida, M. V. & Tsaftaris, S. A. Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans. Med. Imaging 37, 803–814 (2017).

    Article  Google Scholar 

  13. Lee, D., Kim, J., Moon, W.-J. & Ye, J. C. Collagan: collaborative gan for missing image data imputation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2487–2496 (IEEE, 2019).

  14. Havaei, M., Guizard, N., Chapados, N. & Bengio, Y. Hemis: hetero-modal image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 469–477 (Springer, 2016).

  15. Dorent, R., Joutard, S., Modat, M., Ourselin, S. & Vercauteren, T. Hetero-modal variational encoder–decoder for joint modality completion and segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 74–82 (Springer, 2019).

  16. Varsavsky, T., Eaton-Rosen, Z., Sudre, C. H., Nachev, P. & Cardoso, M. J. Pimms: permutation invariant multi-modal segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support 201–209 (Springer, 2018).

  17. Jaderberg, M. et al. Spatial transformer networks. In Advances in Neural Information Processing Systems 2017–2025 (NIPS, 2015).

  18. Wang, F. et al. Residual attention network for image classification. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 3156–3164 (2017).

  19. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 7132–7141 (IEEE, 2018).

  20. Roy, A. G., Siddiqui, S., Pölsterl, S., Navab, N. & Wachinger, C. ‘Squeeze & excite’ guided few-shot segmentation of volumetric images. Med. Image Anal. 59, 101587 (2020).

    Article  Google Scholar 

  21. Roy, A. G., Navab, N. & Wachinger, C. Recalibrating fully convolutional networks with spatial and channel ‘squeeze and excitation’ blocks. IEEE Trans. Medical Imaging 38, 540–549 (2018).

    Article  Google Scholar 

  22. Rickmann, A.-M., Roy, A. G., Sarasua, I., Navab, N. & Wachinger, C. ‘Project & excite’ modules for segmentation of volumetric medical scans. In International Conference on Medical Image Computing and Computer-Assisted Intervention 39–47 (Springer, 2019).

  23. Wang, X., Cai, Z., Gao, D. & Vasconcelos, N. Towards universal object detection by domain attention. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 7289–7298 (IEEE, 2019).

  24. Gomariz, A. et al. Quantitative spatial analysis of haematopoiesis-regulating stromal cells in the bone marrow microenvironment by 3D microscopy. Nat. Commun. 9, 1–15 (2018).

    Article  Google Scholar 

  25. Gomariz, A., Isringhausen, S., Helbling, P. M. & Nombela-Arrieta, C. Imaging and spatial analysis of hematopoietic stem cell niches. Ann. N. Y. Acad. Sci. 1466, 5–16 (2019).

    Article  Google Scholar 

  26. Acar, M. et al. Deep imaging of bone marrow shows non-dividing stem cells are mainly perisinusoidal. Nature 526, 126–130 (2015).

    Article  Google Scholar 

  27. Coutu, D. L., Kokkaliaris, K. D., Kunz, L. & Schroeder, T. Three-dimensional map of nonhematopoietic bone and bone-marrow cells and molecules. Nat. Biotechnol. 35, 1202–1210 (2017).

    Article  Google Scholar 

  28. Coutu, D. L., Kokkaliaris, K. D., Kunz, L. & Schroeder, T. Multicolor quantitative confocal imaging cytometry. Nat. Methods 15, 39–46 (2018).

    Article  Google Scholar 

  29. Christodoulou, C. et al. Live-animal imaging of native haematopoietic stem and progenitor cells. Nature 578, 278–283 (2020).

    Article  Google Scholar 

  30. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).

    MathSciNet  MATH  Google Scholar 

  31. Kunisaki, Y. et al. Arteriolar niches maintain haematopoietic stem cell quiescence. Nature 502, 637–643 (2013).

    Article  Google Scholar 

  32. Gal, Y. & Ghahramani, Z. Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference (ICLR, 2016).

  33. Kendall, A. & Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems 30 5574–5584 (NIPS, 2017).

  34. Raghu, M., Zhang, C., Kleinberg, J. & Bengio, S. Transfusion: understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems Vol. 32, 3347–3357 (NIPS, 2019).

  35. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (eds. Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097–1105 (NIPS, 2012).

  36. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2016).

  37. Xie, Y. et al. Efficient and robust cell detection: a structured regression approach. Med. Image Anal. 44, 245–254 (2018).

    Article  Google Scholar 

  38. Ren, S., He, K., Girshick, R. & Sun, J. Faster r-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intelligence 39, 1137–1149 (2017).

    Article  Google Scholar 

  39. He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask r-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2017).

  40. Yang, L. et al. NuSeT: a deep learning tool for reliably separating and analyzing crowded cells. PLoS Comput. Biol. 16, e1008193 (2020).

    Article  Google Scholar 

  41. Falk, T. et al. U-net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).

    Article  Google Scholar 

  42. Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous systems (2015); https://www.tensorflow.org/

  43. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR, 2015).

  44. Li, Z., Kamnitsas, K. & Glocker, B. Overfitting of neural nets under class imbalance: analysis and improvements for segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention 402–410 (Springer, 2019).

  45. Gómez-de Mariscal, E. et al. DeepImageJ: a user-friendly plugin to run deep learning models in ImageJ. Preprint at https://www.biorxiv.org/content/10.1101/799270v2 (2019).

  46. Gomariz, A. et al. Marker Sampling and Excite: Deep Learning with Heterogeneous Marker Combinations in Fluorescence Microscopy (CodeOcean, 2021); https://codeocean.com/capsule/8424915/tree/v1

Download references

Acknowledgements

We thank T. Nagasawa for the Cxcl12-GFP reporter mice, T. Yokomizo for the Hlf-tdTomato mouse strain and M. Kurokawa for the Evi1-GFP mouse strain. This work was enabled by funding from the Hasler Foundation (A.G. and O.G.). We also acknowledge the support from the Swiss National Science Foundation (SNSF) 179116 (O.G.) and 310030_185171 (C.N.-A.), the Swiss Cancer Research Foundation KFS-3986-08-2016 (C.N.-A.), the Clinical Research Priority Program ‘ImmunoCure’ of the University of Zurich to (C.N.-A.). We extend our thanks to NVIDIA for their GPU support.

Author information

Authors and Affiliations

Authors

Contributions

A.G. designed and performed the experiments and drafted the manuscript. T.P. and O.G. provided key methodological ideas and input. A.G., P.M.H., S.I. and U.S. conducted data acquisition, manual labelling of the images, and helped interpret bone marrow analysis in the biological context. T.P., O.G. and C.N.-A. critically revised the text. A.G., C.N.-A. and O.G. conceived this project. This work was jointly directed by C.N.-A. for the biological and data acquisition aspects and O.G. for computer-scientific and methodological aspects. All authors discussed the results and provided feedback to the writing.

Corresponding author

Correspondence to Alvaro Gomariz.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review informationNature Machine Intelligence thanks Shalin Mehta and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Analysis of attention parameters in ME modules for sinusoids (left) and arteries (right).

We estimate recalibration strength by calculating cosine distances between the ME attention subnetwork outputs obtained for each of the possible input marker combinations. Results are represented as the mean of all such pair-wise distances between all possible marker combinations, at a given layer where ME is placed, with the bars depicting the standard deviations of these distances. Using the colored network layers shown in Fig. 3b, Encoder layers correspond to the network layers in blue, Decoder to the layers in green, and Bottleneck to the yellow layers. The numbers next to each layer indicate the resolution level, where 0 corresponds to the highest (original resolution) and 3 to the lowest (that is, right before and after the bottleneck, for the encoder and decoder, respectively). It can be seen in this representation that recalibration strength is higher in layers with higher resolution, especially near the input of the network. This observation may indicate that high resolution layers of the network focus on effectively combining features from available markers, and in this way create shared abstract features that are common across markers for subsequent processing in lower resolution layers.

Extended Data Fig. 2 Example images for the qualitative assessment of the segmentation of bone marrow images employed for the quantification of vasculature.

Input images contain different combinations of markers shown as an overlay of different colors. The white rectangle within the input images represents the size of the output image when processing with a neural network. White arrows depict inference with the MS-ME model. Different colors are employed in the output images to show the different predicted classes. Since the tissue class overlaps with the other two, its contour is used instead for visual purposes.

Extended Data Fig. 3 Bone marrow volume ratio occupied by sinusoids.

This volume is compared in both diaphysis (DIA) and metaphysis (META) when segmenting them with the morphological image processing (MIP) algorithm previously proposed (n = 12 for both DIA and META) and with our MS-ME method proposed here (n = 61 for DIA, n = 24 for META).

Extended Data Fig. 4 Effect of marker dropout rate rdrop in MS.

F1-score of MS models with different rdrop evaluated on the sinusoids (left) and arteries (right) relative to the proposed MS with rdrop = 0.5. (a) Evaluation for all 31 possible marker combinations (n = 124). Whereas some rdrop ≠ 0.5 produce a slightly superior F1-score for sinusoids, rdrop = 0.5 is the best option for arteries and overall. (b) Median relative F1-score for models evaluated on combinations of specific numbers of markers, each represented by a different color for the different rdrop (\(n=\frac{K!}{(K-k)!k!}\), where K is the number of markers available, and k the number of markers considered for each evaluation). Smaller rdrop are shown to be beneficial for combinations of more markers, and vice-versa. However, this trend becomes noisier for rdrop > 0.5, as illustrated with the gray dashed line. This effect can be due to the decrease in markers observed over time, although it is a question worth of further investigation in future work.

Extended Data Fig. 5 Effect of bias term on ME module.

F1-score of MS-ME model where the bias terms for all ME modules have been removed, relative to the proposed MS-ME model with bias across all marker combinations and cross-validation steps.

Extended Data Fig. 6 Training evolution with our proposed MS-ME model.

The weighted cross-entropy loss (top) and the F1-score (bottom) are shown across epochs for the training (blue) and validation (orange) sets, both for models trained for segmentation of sinusoids (left) and arteries (right). The red dashed line marks the epoch at which we choose the model, based on the highest validation F1-score.

Extended Data Fig. 7 Illustration of the image tiling pipeline employed to create suitable patches for CNNs.

An example of a slice within the 3D image frame is shown in the upper part using Imaris (Bitplane AG). That slice is decomposed in patches as illustrated in the lower part. Each output patch (red) is smaller than their corresponding input patch (cyan) due to the convolutional operations in CNNs. We position the output patches next to each other without overlap in order to avoid padding artifacts in the application of CNNs. Instead, zero padding is only applied along the borders of the whole slice (area with white stripes). When an overlap between output patches cannot be avoided to fill the slice (area with green stripes), the average of the different patches in that region is used.

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gomariz, A., Portenier, T., Helbling, P.M. et al. Modality attention and sampling enables deep learning with heterogeneous marker combinations in fluorescence microscopy. Nat Mach Intell 3, 799–811 (2021). https://doi.org/10.1038/s42256-021-00379-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00379-y

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing