Abstract
Magnetoencephalography (MEG) signals have demonstrated their practical application to reading human minds. Current neural decoding studies have made great progress to build subject-wise decoding models to extract and discriminate the temporal/spatial features in neural signals. In this paper, we used a compact convolutional neural network—EEGNet—to build a common decoder across subjects, which deciphered the categories of objects (faces, tools, animals, and scenes) from MEG data. This study investigated the influence of the spatiotemporal structure of MEG on EEGNet’s classification performance. Furthermore, the EEGNet replaced its convolution layers with two sets of parallel convolution structures to extract the spatial and temporal features simultaneously. Our results showed that the organization of MEG data fed into the EEGNet has an effect on EEGNet classification accuracy, and the parallel convolution structures in EEGNet are beneficial to extracting and fusing spatial and temporal MEG features. The classification accuracy demonstrated that the EEGNet succeeds in building the common decoder model across subjects, and outperforms several state-of-the-art feature fusing methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alexander CA, Halgren E, Marinkovic K, Cash SS (2011) Decoding word and category-specific spatiotemporal representations from MEG and EEG. Neuroimage 54(4):3028–3039. https://doi.org/10.1016/j.neuroimage.2010.10.073
Almeida J, Mahon BZ, Caramazza A (2010) The role of the dorsal visual processing stream in tool identification. Psychol Sci 21(6):772–778
Bai L, Ma H, Huang Y, Luo Y (2005) The development of native Chinese affective picture system–a pretest in 46 college students. Chin Ment Health J. https://doi.org/10.1016/j.molcatb.2005.02.001
Blair K, Marcos PG, Hyung-Suk K, Norcia AM, Patrick S, Joseph N (2015) A representational similarity analysis of the dynamics of object processing using single-trial EEG classification. PLoS ONE. https://doi.org/10.1371/journal.pone.0135697
Cetiner I, Var AA, Cetiner H (2016) Classification of knot defect types using wavelets and KNN. Elektron Elektrotech 22(6):67–72. https://doi.org/10.5755/j01.eie.22.6.17227
Combrisson E, Jerbi K (2015) Exceeding chance level by chance: the caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods 250:126–136
Contini EW, Wardle SG, Carlson TA (2017) Decoding the time-course of object recognition in the human brain: from visual features to categorical decisions. Neuropsychologia. https://doi.org/10.1016/j.neuropsychologia.2017.02.013
Contini EW, Goddard E, Grootswagers T, Williams M, Carlson T (2020) A humanness dimension to visual object coding in the brain. Neuroimage. https://doi.org/10.1016/j.neuroimage.2020.117139
Dalponte M, Bovolo F, Bruzzone L (2007) Automatic selection of frequency and time intervals for classification of EEG signals. Electron Lett 43:1406–1408
Dash D, Ferrari P, Heitzman D, Wang J (2019) Decoding speech from single trial MEG signals using convolutional neural networks and transfer learning. In: International conference of the IEEE engineering in medicine and biology society (EMBC)
Dash D, Ferrari P, Wang J (2020) Decoding imagined and spoken phrases from non-invasive neural (MEG) signals. Front Neurosci. https://doi.org/10.3389/fnins.2020.00290
David FN (2010) Decoding of faces and face components in face-sensitive human visual cortex. Front Psychol. https://doi.org/10.3389/fpsyg.2010.00028
Dima DC, Gavin P, Eirini M, Jiaxiang Z, Singh KD (2018a) Spatiotemporal dynamics in human visual cortex rapidly encode the emotional content of faces. Hum Brain Mapp
Dima DC, Gavin P, Singh KD (2018b) Spatial frequency supports the emergence of categorical representations in visual cortex during natural scene perception. Neuroimage 179:102–116. https://doi.org/10.1016/j.neuroimage.2018.06.033
Felsenstein J (1985) Confidence limits on phylogenies: an approach using the bootstrap. Evolution 39:783–791
Gauthier I, Tarr MJ, Moylan J, Skudlarski P, Gore JC, Anderson AW (2000) The fusiform “face area” is part of a network that processes faces at the individual level. J Cogn Neurosci 12:495–504. https://doi.org/10.1162/089892900562165
Gerson AD, Parra LC, Sajda P (2006) Cortically coupled computer vision for rapid image search. IEEE Trans Neural Syst Rehabil Eng 14(2):174–179. https://doi.org/10.1109/TNSRE.2006.875550
Gross J (2019) Magnetoencephalography in cognitive neuroscience: a primer. Neuron 104(2):189–204. https://doi.org/10.1016/j.neuron.2019.07.001
Horikawa T, Tamaki M, Miyawaki Y, Kamitani Y (2013) Neural decoding of visual imagery during sleep. Science 340(6132):639–642
Huang Z, Yu T (2019) Cross-subject MEG decoding using 3D convolutional neural networks. In: 2019 world robot conference symposium on advanced robotics and automation
Huang W, Xue Y, Hu L, Liuli H (2020) S-EEGNet: electroencephalogram signal classification based on a separable convolution neural network with bilinear interpolation. IEEE Access. https://doi.org/10.1109/ACCESS.2020.3009665
Johnson RW (1998) An introduction to the bootstrap. Teach Stat
Jonas J, Brissart H, Hossu G, Colnat-Coulbois S, Maillard L (2017) A face identity hallucination (palinopsia) generated by intracerebral stimulation of the face-selective right lateral fusiform cortex. Cortex 99:296–310. https://doi.org/10.1016/j.cortex.2017.11.022
Kamitani Y, Tong F (2005) Decoding the visual and subjective contents of the human brain. Nat Neurosci 8(5):679–685. https://doi.org/10.1038/nn1444
Kay KN, Gallant JL (2009) I can see what you see. Nat Neurosci 12(3):245. https://doi.org/10.1038/nn0309-245
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International joint conference on artificial intelligence
Kostas D, Pang EW, Rudzicz F (2019) Machine learning for MEG during speech tasks. Sci Rep. https://doi.org/10.1038/s41598-019-38612-9
Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2016) EEGNet: a compact convolutional network for EEG-based brain-computer interfaces. J Neural Eng. https://doi.org/10.1088/1741-2552/aace8c
Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436. https://doi.org/10.1038/nature14539
Lee SH, Lee M, Lee SW (2020) Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication. IEEE Trans Neural Syst Rehabil Eng 28:2647–2659
Liu C, Kang Y, Zhang L, Zhang J (2020a) Rapidly decoding image categories from MEG data using a multivariate short-time FC pattern analysis approach. IEEE J Biomed Health Inform. https://doi.org/10.1109/JBHI.2020.3008731
Liu C, Li Y, Song S, Zhang J (2020b) Decoding disparity categories in 3-dimensional images from fMRI data using functional connectivity patterns. Cogn Neurodynamics. https://doi.org/10.1007/s11571-019-09557-6
Lopez-Martin M, Carro B, Sanchez-Esguevillas A (2019) IoT type-of-traffic forecasting method based on gradient boosting neural networks. Futur Gener Comput Syst. https://doi.org/10.1016/j.future.2019.12.013
Lopez-Martin M, Nevado A, Carro B (2020) Detection of early stages of Alzheimer’s disease based on MEG activity with a randomized convolutional neural network. Artif Intell Med 107:101924. https://doi.org/10.1016/j.artmed.2020.101924
Marti S, Dehaene S (2017) Discrete and continuous mechanisms of temporal selection in rapid visual streams. Nat Commun. https://doi.org/10.1038/s41467-017-02079-x
Mh A, Mp B, Jc C, Cf D, Wj A, Hba E, Bw F (2020) I tried a bunch of things: the dangers of uneXpected overfitting in classification of brain data. Neurosci Biobehav Rev 119:456–467
Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science. https://doi.org/10.1126/science.1152876
Muukkonen I, Ölander K, Numminen J, Salmela VR (2020) Spatio-temporal dynamics of face perception. Neuroimage. https://doi.org/10.1101/550038
Norman KA, Polyn SM, Detre GJ, Haxby JV (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn Sci 10(9):424–430. https://doi.org/10.1016/j.tics.2006.07.005
Okada T, Tanaka S, Nakai T, Nishizawa S, Konishi J (2000) Naming of animals and tools: a functional magnetic resonance imaging study of categorical differences in the human brain areas commonly used for naming visually presented objects. Neurosci Lett 296:33–36. https://doi.org/10.1016/S0304-3940(00)01612-8
Philiastides MG (2006) Neural representation of task difficulty and decision making during perceptual categorization: a timing diagram. J Neurosci 26(35):8965–8975. https://doi.org/10.1523/jneurosci.1655-06.2006
Qadri S, Dave H, Das R, Alick-Lindstrom S (2021) Beyond the wada: an updated approach to pre-surgical language and memory testing. Epilepsy Res
Qin Y et al (2016) Classifying four-category visual objects using multiple ERP components in single-trial ERP. Cogn Neurodyn 10:275–285. https://doi.org/10.1007/s11571-016-9378-0
Ramkumar P, Jas M, Pannasch S, Hari R, Parkkonen L (2013) Feature-specific information processing precedes concerted activation in human visual cortex. J Neurosci 33(18):7691–7699. https://doi.org/10.1523/JNEUROSCI.3905-12.2013
Riyad M, Khalil M, Adib A (2020) MI-EEGNET: a novel convolutional neural network for motor imagery classification. J Neurosci Methods. https://doi.org/10.1016/j.jneumeth.2020.109037
Sakuraba S, Sakai S, Yamanaka M, Yokosawa K, Hirayama K (2012) Does the human dorsal stream really process a category for tools? J Neurosci 32(11):3949–3953. https://doi.org/10.1523/JNEUROSCI.3973-11.201
Seeliger K, Fritsche M, Güçlü U, Schoenmakers S, Gerven M (2017) Convolutional neural network-based encoding and decoding of visual object recognition in space and time. Neuroimage. https://doi.org/10.1016/j.neuroimage.2017.07.018
Sen S, Daimi SN, Watanabe K, Bhattacharya J, Saha G (2018) A machine learning approach to decode mental states in bistable perception. In: 2017 international conference on information technology (ICIT)
Simanova I, Gerven MV, Oostenveld R, Hagoort P (2012) Identifying object categories from event-related EEG: toward decoding of conceptual representations. PLoS ONE 5(12):e14465. https://doi.org/10.1371/journal.pone.0014465
Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM (2011) Brainstorm: a user-friendly application for MEG/EEG analysis. Comput Intell Neurosci. https://doi.org/10.1155/2011/879716
Taulu S, Simola J (2006) Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Phys Med Biol 51(7):1759–1768. https://doi.org/10.1289/ehp.114-a403a
Vahid A, Bluschke A, Roessner V, Stober S, Beste C (2019) Deep learning based on event-related EEG differentiates children with ADHD from healthy controls. J Clin Med 8(7):1055. https://doi.org/10.3390/jcm8071055
Vahid A, Mückschel M, Stober S, Stock AK, Beste C (2020) Applying deep learning to single-trial EEG data provides evidence for complementary theories on action control. Commu Biol. https://doi.org/10.1038/s42003-020-0846-z
Van de Nieuwenhuijzen ME, Backus AR, Bahramisharif A, Doeller CF, Jensen O, Van Gerven MA (2013) MEG-based decoding of the spatiotemporal dynamics of visual category perception. Neuroimage 83:1063–1073. https://doi.org/10.1016/j.neuroimage.2013.07.075
Wang C, Xiaoping HU, Li Y, Shi X, Jiacai Z (2011) Spatio-temporal pattern analysis of single-trial EEG signals recorded during visual object recognition. Sci China 54(012):2499–2507. https://doi.org/10.1007/s11432-011-4507-1
Wang C, Xiong S, Hu X, Yao L, Zhang J (2012) Combining features from ERP components in single-trial EEG for discriminating four-category visual objects. J Neural Eng 9(5):056013. https://doi.org/10.1088/1741-2560/9/5/056013
Willett FR, Avansino DT, Hochberg LR, Henderson JM, Shenoy KV (2020) High-performance brain-to-text communication via imagined handwriting. BioRxiv https://doi.org/10.1101/2020.07.01.183384
Zhang J, Sudre G, Xin L, Wei W, Bagic A (2011) Clustering linear discriminant analysis for MEG-based brain computer interfaces. IEEE Trans Neural Syst Rehabil Eng 19(3):221–231. https://doi.org/10.1109/TNSRE.2011.2116125
Zheng L, Liao P, Shen L, Sheng J, Gao JH (2019) EMS-Net: a deep learning method for autodetecting epileptic magnetoencephalography spikes. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2019.2958699
Zubarev I, Zetter R, Halme HL, Parkkonen L (2019) Adaptive neural network classifier for decoding MEG signals. Neuroimage. https://doi.org/10.1016/j.neuroimage.2019.04.068
Acknowledgements
This work is funded by the General Program (61977010) of the Nature Science Foundation of China. The authors would also like to thank all anonymous participants of the MEG experiments.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Shi, R., Zhao, Y., Cao, Z. et al. Categorizing objects from MEG signals using EEGNet. Cogn Neurodyn 16, 365–377 (2022). https://doi.org/10.1007/s11571-021-09717-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11571-021-09717-7