Multimodal Fusion of fMRI and EEG for Cognitive State Analysis using Graph Neural Networks (GNNs)
DOI:
https://doi.org/10.31838/NJAP/08.02.07Keywords:
Multimodal Fusion, Cognitive State, Graph Neural Networks (GNNs), Electroencephalography (EEG).Abstract
The integration of multimodal neuroimaging data has emerged as a powerful approach to understanding brain function and cognitive states. Functional Magnetic Resonance Imaging (fMRI) offers high spatial resolution, whereas Electroencephalography (EEG) provides high temporal resolution. When these two technologies are combined, both are very useful for analyzing cognitive states. However, it remains very challenging to integrate these diverse types of data, as they have distinct time scales, spatial dimensions, and signal properties. In this study, we present a novel GNN-based framework that leverages EEG and fMRI data to enhance cognitive state categorization. This approach builds brain graphs that are unique to each individual. The nodes in these graphs represent brain regions or EEG channels, and the edges indicate the interactions and connections between different parts of the brain. We employ a multimodal feature embedding technique to extract additional information from fMRI and EEG. Following that, we employ graph-based convolution and pooling operations to develop hierarchical models of brain activity. This research tested the proposed GNN model on a benchmark dataset for cognitive tasks. It is easier to understand and more accurate than traditional deep learning (DL) and machine learning (ML) approaches. The model also demonstrates how different cognitive states significantly impact brain connections, a finding that is particularly useful for neuroscience. The findings demonstrate that graph-based multimodal fusion is effective, enabling the development of novel techniques for more accurate monitoring of brain states in clinical and neuroergonomic environments. The proposed approach lays the groundwork for future research in multimodal brain network modeling, enabling the better comprehension and decoding of cognitive states in real-time using robust and scalable architectures.
References
1. Huang, Z., Kosan, M., Medya, S., Ranu, S., & Singh, A. (2023, February). Global Counterfactual Explaner for Graph Neural Networks. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining (pp. 141-149).
2. Woźniak, M., Siłka, J., &Wieczorek, M. (2023). Deep Neural Network Correlation Learning Mechanism for CT Brain Tumor Detection. Neural Computing and Applications, 35(20), 14611-14626.
3. Niu, W., Ma, C., Sun, X., Li, M., & Gao, Z. (2023). A brain network analysis-based double-way deep neural network for emotion recognition. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31, 917-925.
4. Wang, J., Li, H., Qu, G., Cecil, K. M., Dillman, J. R., Parikh, N. A., & He, L. (2023). Dynamic Weighted Hypergraph Convolutional Network for Brain Functional Connectome Analysis. Medical image analysis, 87, 102828.
5. Wu, Q., Chen, Y., Yang, C., & Yan, J. (2023). Energy-based out-of-distribution detection for graph neural networks. arXiv preprint arXiv:2302.02914.
6. Chakraborty, B., & Mukhopadhyay, S. (2023). Heterogeneous recurrent spiking neural network for spatio-temporal classification. Frontiers in Neuroscience, 17, 994517.
7. Linka, K., Pierre, S. R. S., & Kuhl, E. (2023). Automated model discovery for the human brain using constitutive artificial neural networks. ActaBiomaterialia, 160, 134-151.
8. Nakra, A., &Duhan, M. (2023). Deep Neural Network with Harmony Search-Based Optimal Feature Selection for EEG Signal Classification in Motor Imagery. International Journal of Information Technology, 15(2), 611-625.
9. Chen, Z., Qing, J., & Zhou, J. H. (2023). Cinematic mindscapes: High-quality video reconstruction from brain activity. Advances in Neural Information Processing Systems, 36, 24841-24858.
10. Tian, S., Zhu, R., Chen, Z., Wang, H., Chattun, M. R., Zhang, S., ... & Lu, Q. (2023). Prediction of suicidality in bipolar disorder using variability of intrinsic brain activity and machine learning. Human brain mapping, 44(7), 2767-2777.
11. Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., ... & Abraham, T. (2023). Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 24705-24728.
12. Lu, Y., Du, C., Zhou, Q., Wang, D., & He, H. (2023, October). Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 5899-5908).
13. Singh, P., Pandey, P., Miyapuram, K., & Raman, S. (2023, June). EEG2IMAGE: image reconstruction from EEG brain signals. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
14. Gao, Y., Jia, B., Houston, M., & Zhang, Y. (2023). Hybrid EEG-fNIRS brain computer interface based on common spatial pattern by using EEG-informed general linear model. IEEE Transactions on Instrumentation and Measurement, 72, 1-10.
15. Mostafavi, A., Cruz-Garza, J. G., & Kalantari, S. (2023). Enhancing lighting design through the investigation of illuminance and correlated colortemperatures' effects on brain activity: An EEG-VR approach. Journal of Building Engineering, 75, 106776.
16. Zheng, K., Yu, S., & Chen, B. (2024). Ci-gnn: A Granger causality-inspired graph neural network for interpretable brain network-based psychiatric diagnosis. Neural Networks, 172, 106147
