Disentangled Multimodal Spatiotemporal Learning for Hybrid EEG-fNIRS Brain-Computer Interface

The hybrid EEG-fNIRS Brain-computer interface (BCI) combines the high temporal resolution of electroencephalography (EEG) with the high spatial resolution of functional near-infrared spectroscopy (fNIRS) to enable comprehensive brain activity detection. However, integrating these modalities to obtain highly discriminative features remains challenging. Most existing methods fail to effectively capture the spatiotemporal coupling features and correlations between EEG and fNIRS signals. Furthermore, these methods adopt a holistic learning paradigm for the representation of each modality, leading to unrefined and redundant multimodal representations. To address these challenges, we propose a disentangled multimodal spatiotemporal learning (DMSL) method for hybrid EEG-fNIRS BCI systems, which simultaneously performs multimodal spatiotemporal coupling and disentangled representation learning within a unified framework. Specifically, DMSL utilizes a compact convolutional module with one-dimensional temporal and spatial convolution layers to extract complex spatiotemporal patterns from each modality and introduces a multimodal attention interaction module to comprehensively capture the inter-modality correlations, enhancing the representations for each modality. Subsequently, DMSL designs an adaptive multi-branch graph convolutional module based on reconstructed channels to effectively capture the spatiotemporal coupling features, incorporating modality consistency and disparity constraints to disentangle common and modality-specific representations for each modality. These disentangled representations are finally adaptively fused to perform different task predictions. The proposed DMSL demonstrates state-of-the-art performance on publicly available datasets for mental arithmetic, motor imagery, and emotion recognition tasks, exceeding the best baselines by 2.34%, 0.59%, and 1.47%, respectively. These results demonstrate the effectiveness of DMSL in improving EEG-fNIRS decoding and its strong generalization ability in BCI applications.