Caveats of Deep Learning for BCI Diagnosis in DOC

Deep learning models have met with remarkable success in various domains due to their ability to extract hierarchical features from high-dimensional data. However, these advantages may not directly apply to electroencephalography (EEG)-based Brain-Computer Interface (BCI) applications for diagnosing Disorders of Consciousness (DOC), where datasets are usually small, heterogeneous, and highly variable across patients and sessions. Here, we compare three EEG Deep Neural Network (DNN) models—EEGNet, DeepConvNet, EEGConformer—with a shallow Linear Discriminant Analysis (LDA) on a DOC dataset. Under within-session leave-one-trial-out (LOTO) evaluation, all DNNs show overoptimistic performance, including above chance results for the coma and all unresponsive wakefulness syndrome (UWS) subjects. Under leave-one-session-out (LOSO) evaluation, in contrast, DNN performance collapses to near random levels. Conversely, session-specific, shallow LDA models are able to avoid these caveats. To interpret these effects, we propose different hypotheses involving the subject-specificity and overall low separability of Sensorimotor Rhythms (SMR) features in DOC, as well as indirect data leakage and/or overfitting to neural correlates of behaviour not related to command following and motor imagery or attempt.