EEG-fNIRS Data Fusion: Techniques, Applications, and Future Directions in Biomedical Research

Abstract

This comprehensive review explores data fusion techniques for integrating electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals to advance biomedical research and clinical applications. It covers the fundamental principles of these complementary neuroimaging modalities, detailed methodological approaches for multimodal fusion, practical strategies for optimizing data quality, and rigorous validation frameworks. Targeted at researchers, scientists, and drug development professionals, the article highlights how EEG-fNIRS fusion overcomes individual modality limitations by combining millisecond temporal resolution with improved spatial localization. Through clinical case studies and emerging research applications, we demonstrate how this integrated approach provides richer insights into brain function, enhances diagnostic capabilities, and accelerates therapeutic development for neurological and psychiatric disorders.

Understanding EEG and fNIRS: Complementary Neuroimaging Modalities

The human brain operates through two primary, interconnected physiological processes: rapid electrical activity from neuronal firing and slower hemodynamic responses that deliver energy resources. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have emerged as complementary neuroimaging techniques that capture these distinct phenomena directly related to neural function [1]. EEG measures the electrical potentials generated by synchronized pyramidal neuron activity, offering millisecond-level temporal resolution to track immediate brain dynamics [1]. In contrast, fNIRS detects hemodynamic changes by measuring concentration changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in the blood, providing better spatial localization of active brain regions [2] [3]. Their integration creates a powerful multimodal approach for studying brain function, leveraging the neurovascular coupling mechanism that links neuronal electrical activity to subsequent vascular responses [1].

Table 1: Fundamental Comparison of EEG and fNIRS Neuroimaging Modalities

Characteristic	EEG (Electrical Activity)	fNIRS (Hemodynamic Response)
Measured Parameter	Electrical potentials from synchronized post-synaptic activity	Concentration changes of HbO and HbR
Physiological Basis	Direct neuronal electrical activity	Neurovascular coupling
Temporal Resolution	Millisecond level (∼5 ms) [4]	Slower (∼0.1-1 Hz) [2] [1]
Spatial Resolution	Low (several centimeters) [5] [1]	Moderate (∼1-3 cm) [4] [5]
Penetration Depth	Superficial and deep sources (with volume conduction)	Cortical surface (1-3 cm) [2]
Primary Signal Origin	Pyramidal neurons in cortical layers [1]	Cerebral microvasculature (arterioles, capillaries, venules)
Noise Sensitivity	Sensitive to electromagnetic artifacts [6]	Sensitive to systemic physiological noise [7]

Physiological Mechanisms and Signaling Pathways

The Electrophysiological Basis of EEG Signals

Electroencephalography captures the macroscopic temporal dynamics of brain electrical activity through passive measurements of scalp voltages [2]. The recorded EEG signal primarily results from the summation of synchronized post-synaptic potentials in cortical pyramidal neurons [1]. For detectable EEG signals to occur, tens of thousands of pyramidal neurons within a cortical column must fire synchronously, with their dendritic trunks coherently oriented parallel to each other and perpendicular to the cortical surface [1]. This specific anatomical arrangement enables sufficient summation and propagation of electrical signals to the scalp surface where electrodes detect voltage fluctuations [1]. These neural oscillations are categorized into characteristic frequency bands—theta (4-7 Hz), alpha (8-14 Hz), beta (15-25 Hz), and gamma (>25 Hz)—each associated with different brain states and cognitive functions [1].

The Hemodynamic Basis of fNIRS Signals

Functional near-infrared spectroscopy leverages the relative transparency of biological tissues to light in the near-infrared spectrum (650-950 nm) to measure hemodynamic changes associated with neural activity [2] [3]. Within this wavelength range, light penetrates the scalp and skull, reaching the cortical surface where it is predominantly absorbed by the chromophores oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) [3]. The neurovascular coupling mechanism forms the foundation for fNIRS: when neurons become active, they trigger a complex cascade that increases local cerebral blood flow, delivering oxygen and nutrients to support metabolic demands [1] [3]. This hemodynamic response manifests as an initial, brief increase in deoxyhemoglobin concentration (the "initial dip") followed by a more pronounced increase in oxyhemoglobin and a decrease in deoxyhemoglobin concentration [3]. fNIRS systems typically utilize two or more wavelengths to distinguish between HbO and HbR based on their distinct absorption spectra [2].

Neurovascular Coupling: Connecting Electrical and Hemodynamic Domains

Neurovascular coupling represents the critical physiological link between neuronal electrical activity and the hemodynamic responses measured by fNIRS [1]. When neurons fire, they consume energy, triggering a complex signaling cascade that involves astrocytes, neurons, and vascular cells [3]. This process begins with increased oxygen extraction from local blood vessels, creating a transient rise in deoxyhemoglobin [3]. Almost simultaneously, vasoactive signals cause arteriolar dilation, increasing local cerebral blood flow and delivering oxygenated blood that typically overshoots metabolic demands [3]. The resulting hemodynamic response peaks 4-6 seconds after neural activation, creating the characteristic fNIRS signal patterns [8]. This tightly regulated mechanism forms the theoretical basis for integrating EEG and fNIRS, as it connects the direct electrical measurements of EEG with the indirect metabolic-hemodynamic measurements of fNIRS [1].

Figure 1: Signaling Pathway Linking Neural Activity to Measurable Signals. The diagram illustrates how initial neural electrical activity triggers neurovascular coupling, leading to measurable EEG and fNIRS signals through different physiological pathways.

Experimental Protocols for Simultaneous EEG-fNIRS Recording

Hardware Integration and Setup

Successful simultaneous EEG-fNIRS recording requires careful hardware integration to ensure signal quality and synchronization. Two primary approaches exist for system integration: (1) separate systems synchronized via computer, and (2) unified systems with a single processor for both modalities [5]. While the first approach offers simplicity, the second provides more precise synchronization essential for capturing the temporal relationship between electrical and hemodynamic responses [5]. For helmet design, researchers typically use flexible EEG electrode caps as a foundation, creating punctures at specific locations to accommodate fNIRS probe fixtures [5]. Customized helmets using 3D printing or cryogenic thermoplastic sheets offer better fit and more consistent optode placement across subjects [5]. The integration must maintain proper source-detector distances (typically 2.5-3.5 cm for adults) for fNIRS while ensuring good electrode-scalp contact for EEG [2] [3]. Optodes and electrodes should be co-registered to anatomical landmarks (nasion, inion, preauricular points) using a 3D digitizer for accurate spatial localization [9].

Table 2: Research Reagent Solutions and Essential Materials

Item Category	Specific Examples	Function/Purpose
fNIRS System Components	Continuous-wave NIRS system (e.g., Hitachi ETG-4100), optical fibers, laser diodes/LEDs (695 & 830 nm) [9]	Generate and detect NIR light to measure HbO/HbR concentration changes
EEG System Components	EEG amplifier system (e.g., BrainAMP), Ag/AgCl electrodes, electrolyte gel [5]	Measure electrical potentials from scalp surface
Integration Materials	Custom integration caps, 3D-printed helmet fixtures, thermoplastic sheets [5]	Co-register EEG and fNIRS components with consistent geometry
Auxiliary Equipment	3D magnetic space digitizer (e.g., Polhemus Fastrak), response recording devices [9]	Record spatial coordinates of optodes/electrodes; capture behavioral data
Software Tools	MNE-Python, NIRS processing toolboxes, custom analysis scripts [8]	Preprocess, synchronize, and analyze multimodal data

Protocol for Motor Execution, Observation, and Imagery Paradigm

The motor execution, observation, and imagery paradigm provides an excellent experimental framework for studying the action observation network (AON) and comparing electrical and hemodynamic responses across different motor conditions [9]. This protocol employs a block design with three conditions: (1) Motor Execution (ME): participants physically perform a motor task (e.g., grasping and moving a cup) in response to an "your turn" auditory cue; (2) Motor Observation (MO): participants observe an experimenter performing the same motor task following a "my turn" cue; and (3) Motor Imagery (MI): participants mentally simulate the motor task without physical movement [9]. Each trial begins with a 2-second instruction period, followed by a 5-second task period, and a variable inter-trial interval (10-20 seconds) to allow hemodynamic responses to return to baseline [9]. For simultaneous EEG-fNIRS recording, participants should be seated comfortably facing an experimenter, with the fNIRS optodes placed over sensorimotor and parietal cortices and EEG electrodes arranged according to the international 10-20 system [9].

Figure 2: Experimental Protocol for Motor Paradigm. The workflow outlines the procedure for simultaneous EEG-fNIRS recording during motor execution, observation, and imagery tasks.

Data Preprocessing and Quality Control

Preprocessing simultaneous EEG-fNIRS data requires modality-specific pipelines executed in parallel before multimodal fusion. For fNIRS data, begin by converting raw intensity signals to optical density, then to hemoglobin concentration changes using the modified Beer-Lambert law [8]. Apply bandpass filtering (0.01-0.5 Hz) to remove cardiac pulsation (∼1 Hz) and slow drifts while preserving the hemodynamic response [8]. For EEG data, implement high-pass filtering (0.5 Hz) to remove slow drifts, notch filtering (50/60 Hz) to remove line noise, and artifact removal techniques (e.g., ICA) to eliminate ocular and muscle artifacts [1]. Quality control metrics should include the scalp coupling index (SCI) for fNIRS (recommended threshold >0.5) and signal-to-noise ratio for EEG [8]. Epoch data according to experimental conditions, with appropriate baseline correction (typically -5 to 0 seconds before stimulus onset) and artifact rejection [8]. Finally, synchronize the preprocessed EEG and fNIRS data streams using recorded trigger pulses or shared clock signals [5].

Data Fusion Methodologies and Applications

Fusion Approaches for EEG-fNIRS Integration

Multimodal fusion of EEG and fNIRS data occurs at three primary levels: decision-level, feature-level, and data-level fusion [1]. Decision-level fusion involves processing and classifying each modality separately, then combining the results at the decision stage using methods like majority voting or meta-classifiers [4]. This approach demonstrated a 5-7% improvement in classification accuracy for motor imagery tasks compared to single-modality approaches [4]. Feature-level fusion combines extracted features from both modalities before classification, often employing techniques like canonical correlation analysis (CCA) or structured sparse multiset CCA (ssmCCA) to identify relationships between electrical and hemodynamic features [9] [4]. This method has shown particular promise in identifying shared neural regions associated with the action observation network during motor tasks [9]. Data-level fusion integrates raw or minimally processed data from both modalities, often using joint blind source separation or model-based approaches to identify common underlying components [7].

Applications in Clinical Neuroscience and Brain-Computer Interfaces

The complementary nature of EEG and fNIRS has enabled advanced applications across clinical and research domains. In clinical neuroscience, simultaneous EEG-fNIRS has been applied to study neurological and psychiatric disorders including epilepsy, Alzheimer's disease, stroke, attention-deficit hyperactivity disorder (ADHD), and amyotrophic lateral sclerosis (ALS) [4] [5]. For epilepsy monitoring, EEG provides precise temporal localization of epileptiform activity while fNIRS offers improved spatial localization of the epileptic focus [3]. In brain-computer interfaces (BCIs), hybrid EEG-fNIRS systems have significantly improved classification accuracy for motor imagery tasks compared to unimodal systems [4] [6]. Recent advances include deep learning approaches with cross-modal attention mechanisms (e.g., MBC-ATT framework) that dynamically weight the contribution of each modality based on task demands, further enhancing classification performance for cognitive state decoding [6].

The fundamental differences between electrical activity and hemodynamic responses, rather than presenting a challenge, create a powerful complementary relationship when captured through simultaneous EEG-fNIRS recording. EEG's millisecond temporal resolution provides an exquisite window into the direct electrical consequences of neural processing, while fNIRS's superior spatial localization captures the metabolically coupled hemodynamic responses that support such activity. The neurovascular coupling mechanism binds these phenomena together, offering researchers a more complete picture of brain function than either modality could provide alone. Through appropriate experimental protocols and advanced fusion methodologies, researchers can leverage these complementary strengths to advance our understanding of brain function in both healthy and pathological states, paving the way for more precise diagnostics and innovative therapeutic interventions in clinical neuroscience.

In non-invasive neuroimaging, the trade-off between temporal and spatial resolution is a fundamental concept. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represent two pillars of brain imaging, each with distinct resolution profiles. Temporal resolution refers to the precision with which a technique can measure when neural activity occurs, while spatial resolution describes its ability to pinpoint where in the brain this activity is generated [10] [11]. EEG is renowned for its millisecond-scale temporal resolution, enabling it to capture rapid neural dynamics. In contrast, fNIRS provides better spatial localization of cortical activity by measuring hemodynamic responses, though its temporal resolution is limited to seconds due to the slow nature of blood flow changes [10] [12]. This application note details the comparative strengths and limitations of these modalities and provides practical protocols for their integrated use in multimodal research, framed within the context of data fusion techniques for enhancing spatiotemporal imaging capabilities.

Technical Comparison of EEG and fNIRS

Quantitative Comparison of Resolutions

Table 1: Technical Specifications of EEG and fNIRS

Feature	EEG (Electroencephalography)	fNIRS (Functional Near-Infrared Spectroscopy)
What It Measures	Electrical activity from postsynaptic potentials of cortical neurons [10]	Hemodynamic response (changes in oxygenated and deoxygenated hemoglobin) [10]
Temporal Resolution	High (millisecond scale) [10] [13]	Low (seconds scale) [10] [14]
Spatial Resolution	Low (centimeter-level) [10] [14]	Moderate (better than EEG, but limited to outer cortex) [10] [12]
Depth of Measurement	Cortical surface [10]	Outer cortex (~1–2.5 cm deep) [10]
Key Strength	Capturing rapid cognitive processes (e.g., sensory perception, ERPs) [10]	Localizing sustained cortical activity (e.g., workload, problem-solving) [10]
Primary Limitation	Spatial smearing due to volume conduction [13]	Indirect, slow hemodynamic response and superficial penetration [10] [12]

Fundamental Limitations and Interdependence of Resolutions

The conventional depiction of temporal and spatial resolution as independent axes is an oversimplification. In practice, the factor that limits one resolution often degrades the other [13].

• EEG's Spatial Blurring Distorts Timing: The primary cause of EEG's poor spatial resolution is volume conduction—the blurring of electrical signals as they pass through the skull and other tissues [13]. This same effect distorts the recovered time course of the underlying neural sources at the scalp level. The recorded signal at any single electrode is a mixture of activities from multiple neural sources, which temporally smears their individual latencies. Techniques like the Surface Laplacian transform, which improves spatial resolution by estimating the Current Source Density (CSD), also secondarily provide a more accurate representation of the underlying source time courses [13].
• fNIRS's Slow Response Obscures Spatial Detail: The hemodynamic response measured by fNIRS evolves over several seconds [14]. If two adjacent brain regions are activated in rapid succession (e.g., with a few hundred milliseconds separation), the slow fNIRS signal will merge their responses into a single, spatially blurred activation, thereby degrading the effective spatial resolution for mapping rapid, sequential neural events [14].

Multimodal Fusion of EEG and fNIRS

The complementary strengths of EEG and fNIRS make them ideal candidates for multimodal fusion, which aims to achieve a comprehensive view of brain activity with high spatiotemporal resolution [7] [14]. Fusion can be implemented at different stages of data processing.

Fusion Approaches and Rationale

Table 2: Data Fusion Approaches for EEG and fNIRS

Fusion Stage	Description	Key Techniques	Benefits
Early-Stage Fusion	Raw or pre-processed data from both modalities are combined before feature extraction [15].	Inputting concurrent EEG and fNIRS data into a Y-shaped neural network [15].	Can capture underlying neurovascular coupling and subtle interactions; has shown higher classification performance in some BCI tasks [15].
Data-Level / Symmetric Fusion	Data-driven methods that treat both modalities equally to find shared latent components [7].	Joint Independent Component Analysis (jICA), Canonical Correlation Analysis (CCA) [7].	Can reveal complex, shared neural processes without strong prior assumptions [7].
Model-Based / Asymmetric Fusion	Using one modality to constrain or inform the analysis of the other [14].	Using high-spatial-resolution fNIRS (or DOT) reconstruction as a spatial prior to constrain the EEG source localization inverse problem [14].	Dramatically improves the spatial resolution of reconstructed neuronal sources; allows resolution of spatially close sources activated with small temporal separations (e.g., 50 ms) [14].

Signaling Pathway and Experimental Workflow

The rationale for fNIRS-EEG fusion is grounded in the principle of neurovascular coupling, where neuronal electrical activity triggers a subsequent hemodynamic response. The following diagram illustrates this relationship and a generalized experimental workflow.

Diagram Title: Neurovascular Coupling and Multimodal Fusion Workflow

Experimental Protocols for Multimodal Imaging

Protocol 1: Simultaneous EEG-fNIRS Setup and Data Acquisition

This protocol ensures high-quality, synchronized data collection.

• Equipment and Reagents:

  • Integrated EEG-fNIRS system or two synchronized standalone systems.
  • EEG cap with pre-defined fNIRS-compatible openings or a custom cap allowing co-registration of electrodes and optodes.
  • EEG electrolyte gel and skin prepping supplies (abrasive gel, alcohol wipes).
  • fNIRS optodes and holders.
  • Head measurement tools for International 10-20 system placement.
  • Synchronization hardware (e.g., TTL pulse generator) if systems are separate.

• Procedure:

  • Participant Preparation: Measure the participant's head according to the 10-20 system. Mark landmark positions (nasion, inion, pre-auricular points). Abrade the skin at electrode sites to reduce impedance to below 10 kΩ for high-density EEG or below 50 kΩ for lower-density setups [16].
  • Cap and Sensor Placement: Secure the integrated cap on the participant's head. For EEG, inject electrolyte gel into the electrodes. For fNIRS, place optodes in their holders, ensuring good skin contact without excessive pressure. Use a cap with a tight but comfortable fit to minimize motion artifacts [10].
  • Hardware Synchronization: If using separate systems, establish a synchronization link. A common method is to send a TTL pulse from a stimulus computer to both the EEG and fNIRS acquisition systems at the start of each trial to align the data streams during post-processing [10].
  • Signal Quality Check: Verify EEG signal quality by inspecting for noise and impedance. Check fNIRS signal quality by ensuring a strong detected light intensity and checking for saturated or absent channels.
  • Data Acquisition: Run the experimental paradigm (e.g., motor imagery, cognitive task) while recording synchronized EEG and fNIRS data.

Protocol 2: Joint EEG-fNIRS Source Reconstruction

This protocol uses fNIRS to spatially constrain EEG source analysis, enhancing spatial accuracy [14].

• Equipment and Software:

  • Computational software (e.g., MATLAB, Python).
  • Toolboxes for EEG source analysis (e.g., FieldTrip, BrainStorm) and fNIRS reconstruction (e.g, HOMER, NIRS-Toolbox).
  • Head model (e.g., from MNI template or individual MRI if available).

• Procedure:

  • Pre-processing: Process EEG and fNIRS data through separate, modality-specific pipelines.
    • EEG: Apply band-pass filtering, artifact removal (e.g., ocular, muscle), and re-referencing.
    • fNIRS: Convert raw light intensity to optical density, then to concentration changes in oxy- and deoxy-hemoglobin. Perform motion artifact correction and band-pass filtering to remove physiological noise [12].
  • fNIRS Source Reconstruction (Spatial Prior): Solve the fNIRS inverse problem to reconstruct the 3D distribution of the hemodynamic activity (e.g., Δ[HbO]) on the cortical surface. This provides a high-spatial-resolution map of activated regions [14].
  • EEG Forward Modeling: Construct a head model and calculate the leadfield matrix, which defines how electrical currents from cortical sources project to the EEG electrodes [14].
  • Joint Inversion: Incorporate the fNIRS reconstruction as a spatial prior constraint in the EEG inverse problem. Algorithms like Restricted Maximum Likelihood (ReML) can be used to fuse the high-temporal-resolution EEG data with the high-spatial-resolution fNIRS prior [14].
  • Validation: Validate the reconstructed activity against known neurophysiological principles or task timings.

Protocol 3: Early-Stage Fusion for Brain-Computer Interface (BCI) Classification

This protocol uses a Y-shaped neural network to fuse EEG and fNIRS data at the raw data level for improved classification of mental states [15].

• Equipment and Software:

  • Pre-processed, synchronized EEG-fNIRS dataset.
  • Machine learning environment (e.g., Python with PyTorch/TensorFlow).

• Procedure:

  • Data Preparation: Segment the synchronized EEG and fNIRS data into epochs time-locked to the events of interest (e.g., cue for motor imagery). Normalize the data.
  • Network Architecture:
    • Input Layer: Two separate input branches for EEG data and fNIRS data.
    • Modality-Specific Encoders: The EEG branch can use a architecture like EEGNet to extract temporal and spectral features. The fNIRS branch, due to its lower temporal complexity, may use a simpler network (e.g., the later layers of EEGNet) to extract features from the hemoglobin concentration time series [15].
    • Fusion Layer (Early-Stage): Concatenate the feature maps from both encoders at an early stage in the network, before the final classification layers.
    • Classification Head: The concatenated features are fed into fully connected layers and a softmax output layer to generate the final classification (e.g., left vs. right hand motor imagery) [15].
  • Training: Train the network using an appropriate optimizer and loss function (e.g., cross-entropy loss), validating performance on a held-out test set.
  • Evaluation: Compare the classification accuracy of this early-fusion model against models using only EEG or only fNIRS, or against models using late-fusion (decision-level) strategies.

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 3: Key Materials for Multimodal EEG-fNIRS Research

Item	Function/Description	Example Use Case
Integrated EEG-fNIRS Caps	Head caps with pre-defined placements for both electrodes and optodes, ensuring consistent co-registration across sessions [10].	Essential for any multimodal study to ensure sensors are positioned correctly relative to each other and brain anatomy.
Conductive Electrolyte Gel	Reduces impedance between the scalp and EEG electrodes, facilitating the measurement of electrical potentials.	Standard requirement for obtaining high-quality EEG signals.
Short-Separation fNIRS Channels	fNIRS detectors placed very close (~8 mm) to a source, which are predominantly sensitive to systemic physiological noise in the scalp [7].	Used as regressors in data processing to separate superficial (scalp) from deep (cortical) components of the fNIRS signal, improving signal quality [7].
TTL Pulse Generator / Sync Box	A hardware device that sends precise digital timing pulses to multiple data acquisition systems.	Critical for synchronizing the clocks of separate EEG and fNIRS systems during simultaneous recording.
3D Digitizer	A stylus-based system to record the precise 3D locations of EEG electrodes and fNIRS optodes relative to head landmarks.	Improves the accuracy of source reconstruction by providing exact sensor positions for the forward model.
Surface Laplacian (CSD) Toolbox	Software (e.g., in EEGLAB) that computes the Current Source Density (CSD) from scalp potentials.	Used to reduce the spatial blurring effect of volume conduction in EEG, thereby improving both spatial and temporal resolution of scalp-level data [13].

Neurovascular coupling (NVC) describes the fundamental biological mechanism that creates a tight temporal and regional linkage between neural activity and subsequent changes in cerebral blood flow (CBF). This physiological process ensures that active brain regions promptly receive an increased supply of oxygen and nutrients to meet heightened metabolic demands [17]. The adult human brain constitutes approximately 2% of total body weight yet consumes over 20% of the body's oxygen and glucose at rest, creating a critical dependence on precisely regulated blood flow delivery [17]. The neurovascular unit, composed of vascular smooth muscle cells, neurons, and astrocyte glial cells, forms the anatomical foundation for this sophisticated communication system [17] [18].

The significance of neurovascular coupling extends beyond basic physiology to underpin modern functional neuroimaging techniques. Both functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS) rely on the hemodynamic changes triggered by neural activity to indirectly map brain function [19] [3]. Understanding NVC is therefore paramount for proper interpretation of neuroimaging data across both research and clinical settings. impairments in neurovascular coupling have been implicated in various pathological conditions including Alzheimer's disease, stroke, hypertension, and diabetes, highlighting its clinical relevance [18] [1].

Biological Mechanisms of Neurovascular Coupling

Cellular Components and Signaling Pathways

The neurovascular unit operates through coordinated interactions between three primary cell types: neurons, astrocytes, and vascular smooth muscle cells. Upon neuronal activation, glutamate released from presynaptic terminals activates postsynaptic neurons and surrounding astrocytes [17]. Neurons respond by activating neuronal nitric oxide synthase (nNOS), producing nitric oxide (NO) that directly dilates parenchymal arterioles [17]. Concurrently, astrocytes respond to glutamate through metabotropic glutamate receptors, triggering calcium-dependent production of vasoactive compounds [17] [19].

The signaling cascade in astrocytes involves production of arachidonic acid, which is subsequently metabolized into several vasoactive messengers: prostaglandin E2 (PGE2) via cyclooxygenase-1 (COX-1), epoxyeicosatrienoic acid (EET) via cytochrome P450 (CYP) epoxygenase, and 20-Hydroxyeicosatetraenoic acid (20-HETE) via CYP4A [19]. These molecules collectively modulate vascular tone by acting on arterioles and capillaries. Additionally, GABA interneurons release various vasoactive substances including NO, acetylcholine, neuropeptide Y, and vasoactive intestinal peptide (VIP), which can produce both constrictive and dilatory effects on cerebral microvasculature [17].

Figure 1: Cellular Signaling Pathways in Neurovascular Coupling. This diagram illustrates the primary mechanisms through which neural activity triggers hemodynamic responses. Key pathways include neuronal nitric oxide production, astrocytic vasoactive compound synthesis, and interneuron-mediated regulation.

Hemodynamic Response Characteristics

The canonical hemodynamic response to neural activity follows a characteristic pattern known as "functional hyperemia." During baseline conditions, cerebral blood flow is closely matched to metabolic demands. Upon neural activation, a biphasic response occurs: an initial brief increase in deoxygenated hemoglobin (the "initial dip") followed by a substantial increase in cerebral blood flow that delivers oxygenated hemoglobin beyond metabolic requirements [17] [3]. This overcompensation results in a 4-fold greater increase in CBF relative to the increase in ATP consumption, forming the physiological basis for blood-oxygen-level-dependent (BOLD) contrast used in fMRI [17].

The hemodynamic response profile varies between populations, with neonates and preterm infants demonstrating delayed and less pronounced responses compared to adults [18]. This developmental difference is attributed to ongoing maturation of neurovascular unit components including astrocytes and pericytes during early life [18]. The typical adult response to a stimulus produces a 10-20% increase in cerebral blood flow in the posterior cerebral artery and a 5-8% increase in the middle cerebral artery territory [18].

Quantitative Models and Analysis Frameworks

Mathematical Modeling of Neurovascular Coupling

Computational models of neurovascular coupling integrate complementary data from animal and human studies to create quantitative frameworks for predicting hemodynamic responses. Sten et al. (2023) developed a comprehensive mathematical model that preserves mechanistic behaviors across species by translating cell-specific contributions identified in rodent optogenetics studies to human applications [19]. Their model identifies distinct neuronal subpopulations responsible for different temporal components of the vascular response: NO-producing interneurons mediate the initial rapid dilation, pyramidal neurons sustain the main dilation during prolonged stimuli, and NPY-interneurons contribute to the post-stimulus undershoot [19].

These models enable more accurate interpretation of neuroimaging data by accounting for the complex, non-linear relationships between neural activity and hemodynamic responses. For example, they incorporate the interplay between cerebral metabolic rate of oxygen (CMRO2), cerebral blood flow (CBF), and cerebral blood volume (CBV) that governs the BOLD response [19]. Such quantitative frameworks are particularly valuable for understanding pathophysiological conditions where neurovascular coupling is impaired, including stroke, neurodegenerative diseases, and developmental disorders [18] [19].

Signal Analysis Techniques

Advanced signal processing methods have been developed to characterize neurovascular coupling in both task-based and resting-state paradigms. For spatiotemporal studies involving controlled stimuli, the hemodynamic response function can be modeled using general linear models that account for the typical delay and dispersion of the blood flow response relative to neural activity [7] [18]. Resting-state analyses employ correlation techniques, coherence analysis, and graph theory to identify spontaneous coupling between electrical and hemodynamic fluctuations [18].

Recent methodological advances include the application of Dirichlet distribution parameter estimation to model uncertainty in multimodal data fusion and Dempster-Shafer theory for evidence combination from heterogeneous signal sources [20]. These approaches are particularly valuable for integrating EEG and fNIRS measurements, as they provide formal frameworks for handling the different temporal characteristics and physiological origins of electrical and hemodynamic signals [20] [1].

Table 1: Quantitative Parameters of Neurovascular Coupling

Parameter	Typical Value/Range	Measurement Context	Technical Notes
CBF Response to CO₂	1-6% per mm Hg CO₂ change	Global cerebral hemodynamics	Particularly sensitive in hypercapnic range [17]
Temporal Delay	1-5 seconds post-stimulus	Adult sensory stimulation	Longer delays observed in neonates [18]
Hemodynamic Response Duration	10-20 seconds	Brief sensory stimulus (e.g., finger tapping)	Duration depends on stimulus length and intensity [18]
Spatial Extent of CBF Increase	10-20% in PCA, 5-8% in MCA	Task-activated regions	Varies by vascular territory [18]
Initial Dip Onset	1-2 seconds post-stimulus	BOLD fMRI/fNIRS	Not always reliably detected [3]

Multimodal Integration of EEG and fNIRS

Methodological Rationale and Advantages

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) provides a powerful multimodal approach for investigating neurovascular coupling by simultaneously capturing electrical neural activity and hemodynamic responses [1]. These modalities offer complementary technical properties: EEG delivers millisecond-level temporal resolution of neural oscillations but limited spatial resolution, while fNIRS provides better spatial localization of hemodynamic changes with reasonable temporal resolution (typically up to 10 Hz) [21] [1]. Furthermore, the signals originate from distinct yet coupled physiological processes—EEG primarily reflects postsynaptic potentials of cortical pyramidal neurons, while fNIRS measures hemodynamic changes associated with neurovascular coupling [1].

The practical advantages of combined EEG-fNIRS systems include portability, non-invasiveness, relative tolerance to movement artifacts, and compatibility with naturalistic environments [7] [1]. This makes them particularly suitable for studying brain function in special populations including children, patients with implants, and those requiring bedside monitoring [21] [1]. The built-in validation provided by measuring both electrical and hemodynamic responses to neural activation strengthens the interpretation of observed brain activity patterns [1].

Data Fusion Strategies

Three primary methodological categories have emerged for analyzing concurrent EEG-fNIRS data:

• EEG-informed fNIRS analysis: Using EEG features to guide the analysis of hemodynamic responses, particularly for event-related paradigms where precise timing from EEG can improve modeling of the delayed fNIRS response [1].

• fNIRS-informed EEG analysis: Employing hemodynamic information to constrain source localization of electrical neural activity, addressing EEG's inherent spatial limitations [1].

• Parallel EEG-fNIRS analysis: Processing both modalities independently followed by integrated interpretation, often using advanced fusion techniques such as classifier combination or evidence theory [20] [1].

Deep learning approaches have shown particular promise for multimodal fusion, with recent implementations using dual-scale temporal convolution and hybrid attention modules for EEG feature extraction combined with spatial convolution and gated recurrent units for fNIRS processing [20]. Decision-level fusion using Dempster-Shafer theory has achieved 83.26% accuracy in motor imagery classification, demonstrating the practical benefits of integrated analysis [20].

Figure 2: EEG-fNIRS Integration Workflow. This diagram outlines the major approaches for combining electrophysiological (EEG) and hemodynamic (fNIRS) data, including preprocessing steps, fusion strategies, and application domains.

Experimental Protocols for Neurovascular Coupling Assessment

Stimulus Paradigms and Experimental Design

Well-controlled stimulus paradigms are essential for reliable assessment of neurovascular coupling. In adult populations, standardized motor tasks such as finger tapping reliably activate contralateral motor cortex, producing measurable hemodynamic responses [18]. Visual stimuli using checkerboard patterns or flashing LEDs, and auditory stimuli using tone sequences or speech samples, effectively activate respective sensory cortices [18]. For neonatal and infant populations, flashing LEDs are particularly advantageous as they can be administered during sleep, minimizing movement artifacts [18].

Block-designed experiments, alternating between active and rest conditions, enhance the detection of stimulus-locked hemodynamic responses by improving signal-to-noise ratio through multiple trial averaging [3]. Event-related designs allow for more natural task structures and reduce habituation effects but typically require more sophisticated analysis approaches [3]. The selection of appropriate inter-stimulus intervals is critical, as the hemodynamic response requires 10-20 seconds to return to baseline following a brief stimulus [18].

Protocol Standardization Guidelines

To improve reliability and comparability across studies, standardized assessment protocols for neurovascular coupling should incorporate the following elements:

• Physiological monitoring: Continuous measurement of end-tidal CO₂, blood pressure, and heart rate to account for systemic influences on cerebral hemodynamics [17].

• Artifact handling: Implementation of robust artifact removal techniques, including short-separation channels for fNIRS to correct for superficial scalp blood flow contributions, and advanced algorithms for EEG artifact suppression [7] [21].

• Stimulus characterization: Precise documentation of stimulus parameters (duration, intensity, modality) and timing synchronization between stimulus presentation and data acquisition [18].

• Quality metrics: Establishment of predefined quality thresholds for signal-to-noise ratio, motion artifact contamination, and physiological noise levels for data inclusion [21] [3].

Recent methodological developments include automated software that coalesces repetitive trials into single contours and extracts multiple neurovascular coupling metrics including response latency, peak amplitude, and duration [17]. Such tools also facilitate normalization of responses to dynamic changes in arterial blood gases, which significantly influence the hyperemic response [17].

Research Reagent Solutions and Technical Tools

Table 2: Essential Research Materials for Neurovascular Coupling Studies

Category	Specific Examples	Function/Application	Technical Notes
fNIRS Systems	Continuous-wave NIRS, Time-domain NIRS, Frequency-domain NIRS	Hemodynamic response measurement	CW-NIRS most common due to cost and simplicity [1]
EEG Systems	High-density EEG systems, Amplifiers, Active/passive electrodes	Neural electrical activity recording	Sampling rates typically 256-1024 Hz; higher than fNIRS [1]
Stimulation Equipment	LCD displays for visual patterns, Tactile stimulators, Auditory delivery systems	Controlled stimulus presentation	LED-based visual stimuli suitable for sleeping neonates [18]
Physiological Monitoring	Capnography, Pulse oximetry, Non-invasive BP monitoring	Control for systemic confounders	Essential for normalizing cerebral hemodynamic responses [17]
Analysis Software	Homer2, NIRS-SPM, EEGLAB, FieldTrip, Custom MATLAB scripts	Data processing and fusion	Increasing availability of open-source toolboxes [7] [1]
Computational Modeling Tools	MATLAB with custom scripts, Monte Carlo simulation packages	Quantitative modeling of NVC	Enables integration of animal and human data [19]

Clinical Applications and Future Directions

Neurovascular coupling assessment has significant clinical utility across multiple neurological conditions. In critical care settings, fNIRS monitoring of cerebral oxygenation and autoregulation provides valuable information for managing patients with stroke, traumatic brain injury, and undergoing neurosurgical procedures [3]. The non-invasive nature and portability of fNIRS-EEG systems make them particularly suitable for prolonged monitoring in intensive care units [3].

In neurodegenerative disorders, impaired neurovascular coupling has been identified as a potential early biomarker for Alzheimer's disease and vascular dementia [18] [1]. The combination of fNIRS with EEG offers a practical method for repeated assessment of neurovascular function in these patient populations [1]. Additionally, epilepsy monitoring benefits from simultaneous electrical and hemodynamic recording, as the spread of seizure activity involves complex neurovascular interactions that can be localized through multimodal integration [3].

Future methodological developments are likely to focus on several key areas. First, the creation of high-density whole-head optode arrays with anatomical co-registration will improve spatial accuracy and enable more reliable individual subject analysis [3]. Second, advanced signal processing techniques including vector diagram analysis to detect the initial dip in the hemodynamic response may enhance temporal precision for event detection [3]. Third, the integration of additional physiological parameters such as cytochrome oxidase measurements will provide more comprehensive assessment of metabolic function alongside hemodynamic responses [21] [3].

The expanding application of machine learning and deep learning approaches to multimodal data fusion promises to advance both basic understanding of neurovascular coupling and its clinical applications. These techniques enable more sophisticated pattern recognition in complex datasets and improve real-time classification of brain states for brain-computer interfaces and clinical monitoring systems [20] [3]. As these methodologies mature, standardized protocols for assessing neurovascular coupling will become increasingly important for translating research findings into clinical practice.

{article}

Naturalistic Brain Imaging: Advantages Over fMRI and PET in Real-World Settings

Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) are revolutionizing cognitive neuroscience by enabling brain imaging in naturalistic, real-world settings. This paradigm shift addresses a critical limitation of traditional neuroimaging techniques like functional magnetic resonance imaging (fMRI) and positron emission tomography (PET), which require strict physical constraints and immobilization, thereby compromising ecological validity. This article details the technical advantages of portable fNIRS-EEG systems and provides structured Application Notes and Protocols for their use. Framed within a broader thesis on data fusion techniques, the content provides researchers and drug development professionals with practical methodologies for leveraging multimodal signal fusion to gain deeper, more authentic insights into brain function.

Traditional neuroimaging techniques, particularly fMRI and PET, have been foundational for human neuroscience. Functional MRI (fMRI) explores brain architecture and activity patterns by measuring the blood-oxygen-level-dependent (BOLD) signal, offering high spatial resolution for localizing neural activity across the entire brain [22]. PET imaging relies on the use of radiolabeled tracers to measure metabolic processes, such as glucose utilization. However, a key assumption of these methods is that findings from highly controlled laboratory settings can be generalized to mental processes in real-world scenarios [23]. This assumption is increasingly being challenged.

The pursuit of ecological validity—the extent to which experimental findings reflect real-world behavior and experience—is driving the adoption of naturalistic neuroimaging [23]. This approach utilizes portable technologies like fNIRS and EEG to study the brain in dynamic, interactive contexts. Research has revealed significant differences in brain activation between traditional paradigms and those that approximate real life. For example, the amygdala, often identified as a "fear center" in studies using static threatening images, is not engaged in the same way during dynamic and prolonged fear experiences, suggesting that different neural systems underpin acute versus sustained affective states [23]. Moving beyond the scanner is therefore not merely a technical convenience but a necessity for a complete understanding of brain function.

Comparative Advantages of Portable Modalities

The following table summarizes the key characteristics of fMRI and PET compared to the portable modalities used for naturalistic imaging.

Table 1: Comparative Analysis of Neuroimaging Modalities for Real-World Settings

Feature	fMRI	PET	fNIRS	EEG	fNIRS-EEG Fusion
Spatial Resolution	High (millimeters) [22]	High (millimeters)	Moderate (centimeters) [24]	Low (centimeters) [24]	Enhanced & Complementary [24]
Temporal Resolution	Slow (1-5 seconds) [22]	Very Slow (minutes)	Slow (1-5 seconds)	Very Fast (milliseconds) [24]	Enhanced & Complementary [24]
Portability	No	No	Yes [25] [24]	Yes [24]	Yes [26]
Tolerance to Motion	Low	Low	High [25]	Moderate	High [25]
Measurement Type	Hemodynamic (BOLD) [22]	Metabolic (Glucose)	Hemodynamic (HbO/HbR) [24]	Electrical Activity [24]	Neurovascular Coupling [7]
Key Real-World Advantage	N/A (Scanner-bound)	N/A (Scanner-bound)	Bedside/ecological use [25]	Real-time tracking	Comprehensive brain activity decoding [7]

The fusion of fNIRS and EEG creates a synergistic system that overcomes the limitations of each individual modality. EEG provides excellent temporal resolution to track rapid neural events, while fNIRS offers better spatial resolution for localizing the hemodynamic response associated with neural activity [24]. This combination is particularly powerful for studying complex, real-world cognitive processes and for applications in clinical populations where traditional scanning is impractical [25].

Data Fusion Techniques for EEG and fNIRS

Integrating signals from EEG and fNIRS requires sophisticated data fusion strategies. The choice of fusion stage presents a key research decision, with each level offering distinct advantages.

Table 2: Data Fusion Techniques for Multimodal EEG-fNIRS Data

Fusion Stage	Description	Key Findings & Applications
Early-Stage (Data-Level) Fusion	Raw or pre-processed data from both modalities is combined into a single data structure for input into a model.	Shown to significantly improve classification of motor imagery tasks compared to middle- or late-stage fusion [15].
Feature-Level Fusion	Features are extracted from each modality separately and then concatenated into a combined feature vector.	Direct concatenation has performed on par with the best decision-level techniques in emotion recognition tasks [27].
Decision-Level (Late) Fusion	Each modality is processed through separate models, and the final decisions (e.g., classifications) are combined.	Average-based soft voting has shown strong performance in emotion recognition across valence, arousal, and dominance dimensions [27].

The following diagram illustrates the workflow of a Y-shaped neural network, a common architecture for investigating these fusion strategies:

Multimodal Fusion Network - This Y-shaped network processes EEG and fNIRS signals through separate encoders before fusing them for a unified output.

Application Notes & Experimental Protocols

Application Note 1: Motor Imagery for Neurorehabilitation

Background: Motor imagery (MI)-based neurofeedback is a promising tool for post-stroke motor rehabilitation, aiming to stimulate neuroplasticity in lesioned motor areas [26]. Unimodal neurofeedback has limitations, with approximately 30% of users unable to self-regulate brain activity effectively [26].

Objective: To assess the efficacy of multimodal EEG-fNIRS neurofeedback compared to unimodal (EEG-only or fNIRS-only) NF for upper-limb motor imagery.

Experimental Protocol:

• Participants: 30 right-handed healthy volunteers (as a precursor to clinical trials) [26].
• Equipment & Setup:
  • fNIRS System: Continuous-wave system (e.g., NIRScout XP) with sources and detectors positioned over the sensorimotor cortices to measure changes in oxygenated (HbO) and deoxygenated (HbR) hemoglobin [26].
  • EEG System: 32-channel system (e.g., ActiCHamp) with electrodes positioned according to the 10-10 international system over sensorimotor areas [26].
  • Integrated Cap: A custom cap holding both EEG electrodes and fNIRS optodes to ensure concurrent recording and co-localization [26].
• Procedure:
  • Participants undergo three randomized NF conditions in a single session: EEG-only, fNIRS-only, and combined EEG-fNIRS.
  • Each trial presents a visual cue (e.g., a black arrow) for 2 seconds, indicating which hand to imagine moving.
  • This is followed by a 10-second MI period, where participants perform kinesthetic motor imagery (e.g., imagining opening and closing a hand). During this period, a real-time NF score is calculated and presented as visual feedback (e.g., a moving ball on a gauge).
  • The session includes a rest period of 10-12 seconds between trials [26].
• Data Fusion & Analysis:
  • Real-time Processing: For the fused condition, an NF score is computed by integrating features from both modalities. The EEG feature is typically the event-related desynchronization (ERD) in the mu/beta band over the sensorimotor cortex. The fNIRS feature is the increase in HbO concentration in the contralateral motor cortex [26].
  • Outcome Measures: Primary outcomes include the NF performance score and the amplitude of brain activity modulation in the targeted sensorimotor cortex. Exploratory outcomes include subjective reports of MI vividness and feeling of control over the NF [26].

Table 3: Research Reagent Solutions for Motor Imagery Protocol

Item	Specification / Example	Function in Experiment
fNIRS System	NIRScout XP (NIRx) with 16 detectors, 16 LED sources	Measures hemodynamic response (HbO/HbR) in the cortex during motor imagery.
EEG System	ActiCHamp (Brain Products) 32-channel amplifier	Records electrical brain activity (ERD/ERS) with high temporal resolution.
Integrated Cap	EasyCap (CNX-128) with integrated EEG & fNIRS holders	Ensures stable and co-registered positioning of all sensors on the scalp.
Stimulus Presentation Software	MATLAB with Psychtoolbox or Presentation	Presents visual cues and the real-time neurofeedback metaphor to the participant.
Real-time Processing Platform	Custom software (e.g., via Lab Streaming Layer)	Synchronizes data streams, extracts features, computes NF score, and controls feedback.

Application Note 2: Naturalistic Emotion Recognition

Background: Emotions are complex states that unfold dynamically in real-world contexts. Traditional experiments using static stimuli lack the ecological validity to capture these processes fully [23] [27].

Objective: To develop a personalized multi-modal fNIRS-EEG system for decoding the dynamic trajectories of emotional experiences in naturalistic settings.

Experimental Protocol:

• Stimuli: Use of ecologically valid stimuli such as movie clips known to elicit strong, dynamic emotional responses (e.g., fear, sadness, amusement) [23].
• Equipment & Setup:
  • Similar to Protocol 4.1, using a portable, synchronized fNIRS-EEG system to allow for some participant movement and a more immersive experience.
• Procedure:
  • Participants watch a series of movie clips while fNIRS and EEG data are recorded concurrently.
  • Following each clip, participants provide continuous or discrete self-assessments of their emotional state along dimensions such as valence (pleasantness), arousal (intensity), and dominance [27].
• Data Fusion & Analysis:
  • Feature Extraction: For EEG, features include spectral power in frequency bands of interest (e.g., alpha, beta). For fNIRS, features include the mean and slope of HbO and HbR time series.
  • Fusion and Classification: Implement and compare feature-level (e.g., direct concatenation of features) and decision-level (e.g., soft voting of classifier outputs) fusion techniques [27].
  • Validation: Use machine learning models to classify the self-reported emotional states and compare the classification accuracy of the fused system against unimodal benchmarks.

The workflow for such a naturalistic emotion study is outlined below:

Emotion Decoding Workflow - This protocol uses naturalistic stimuli and fused neuroimaging to decode emotional states.

Discussion and Future Perspectives

The integration of fNIRS and EEG represents a significant step toward truly naturalistic brain imaging, offering a compelling alternative to the scanner-bound environment of fMRI and PET. The synergistic combination of these modalities provides a more comprehensive picture of brain function by capturing complementary aspects of neural activity [24]. This is particularly valuable for tracking the dynamic, context-dependent, and personalized nature of cognitive and affective processes as they occur in real-world environments [23].

Future advancements in this field will likely focus on refining data-driven fusion techniques, such as source-decomposition methods, to better reveal latent neurovascular coupling processes [7]. Furthermore, the development of more miniaturized, robust, and user-friendly hardware will continue to push neuroimaging out of the laboratory and into clinics, homes, and real-world settings, opening unprecedented opportunities for diagnosing, monitoring, and treating neurological and psychiatric disorders [25] [28].

{/article}

Technical and Physical Basis of Signal Acquisition Methods

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a leading approach in non-invasive neuroimaging, capitalizing on the complementary strengths of each modality. This fusion provides a more comprehensive picture of brain activity by combining EEG's millisecond-level temporal resolution with fNIRS's superior spatial localization capabilities [7] [29] [6]. While EEG captures postsynaptic potentials from pyramidal neurons, reflecting electrical brain activity with high temporal precision, fNIRS measures hemodynamic responses associated with neural activity through near-infrared light, providing better spatial resolution and resistance to motion artifacts [29] [6]. This combination is particularly valuable for brain-computer interface (BCI) applications, clinical neurology, and neurorehabilitation research, where understanding both rapid neural dynamics and underlying metabolic processes is essential [7] [29]. The technical and physical foundations of these signal acquisition methods must be thoroughly understood to design effective experiments and implement appropriate data fusion strategies.

Technical Principles and Physical Foundations

Electroencephalography (EEG) Physical Basis

EEG measures electrical potentials generated by the synchronized postsynaptic activity of pyramidal neurons in the cerebral cortex. These electrical signals originate from ionic currents flowing during neuronal excitation and inhibition, creating dipole fields that can be detected on the scalp surface [6]. The conductive properties of biological tissues (scalp, skull, cerebrospinal fluid, and brain) volume-conduct these signals, which typically range from 10 to 100 microvolts in amplitude. EEG electrodes detect voltage differences between active sites and reference points, capturing oscillatory activity across multiple frequency bands (delta, theta, alpha, beta, gamma) that correlate with various brain states and cognitive processes.

Table 1: Key Characteristics of EEG Signal Acquisition

Parameter	Specification	Physiological Basis	Technical Considerations
Physical Signal	Electrical potentials	Postsynaptic potentials of pyramidal neurons	Voltage differences measured at scalp surface
Temporal Resolution	Millisecond level (<100 ms)	Direct measurement of neural firing	Limited only by sampling rate (typically 250-2000 Hz)
Spatial Resolution	~1-3 cm (limited)	Signal smearing by volume conduction	Improved with high-density arrays (64-256 channels)
Frequency Bands	Delta (0.5-4 Hz), Theta (4-8 Hz), Alpha (8-13 Hz), Beta (13-30 Hz), Gamma (30-100 Hz)	Different cognitive states and processes	Filter settings critical for isolating bands of interest
Artifact Sources	Ocular, muscle, cardiac, environmental noise	Non-neural bioelectric activity, external interference	Requires robust artifact detection/correction strategies

Functional Near-Infrared Spectroscopy (fNIRS) Physical Basis

fNIRS utilizes near-infrared light (typically 690-850 nm wavelengths) to measure hemodynamic changes in cortical tissue. The physical principle is based on the modified Beer-Lambert law, which describes light attenuation in scattering media like biological tissue [30]. At these wavelengths, light penetrates biological tissues effectively and is differentially absorbed by oxygenated (HbO) and deoxygenated hemoglobin (HbR), enabling quantification of changes in cerebral blood oxygenation associated with neural activity through neurovascular coupling.

The technique employs optical sources (emitting specific wavelengths) and detectors arranged on the scalp with specific separation distances (typically 3-4 cm), creating measurement channels. The differential pathlength factor (DPF) accounts for light scattering in tissue, while the partial volume factor (PVF) corrects for the fraction of the path that passes through brain tissue versus other tissues [30]. These are often combined into a partial pathlength factor (PPF) for practical application in concentration calculations.

Table 2: Key Characteristics of fNIRS Signal Acquisition

Parameter	Specification	Physiological Basis	Technical Considerations
Physical Signal	Light attenuation	Hemoglobin absorption of NIR light	Measures HbO and HbR concentration changes
Temporal Resolution	~0.1-1.0 seconds	Hemodynamic response delay (neurovascular coupling)	Limited by slow hemodynamic response (5-8 sec peak)
Spatial Resolution	5-10 mm	Limited by source-detector separation (typically 3 cm)	High-density arrays enable tomographic reconstruction (HD-DOT)
Depth Sensitivity	Superficial cortex (1-3 cm)	Light scattering limits penetration	Short-separation channels help correct for superficial artifacts
Artifact Sources	Motion, scalp blood flow, systemic physiology	Non-cerebral hemodynamics, movement	Motion correction algorithms essential

Complementary Nature of EEG and fNIRS

The synergy between EEG and fNIRS stems from their complementary measurement principles and spatiotemporal characteristics. EEG provides direct measurement of neural electrical activity with excellent temporal resolution, ideal for tracking rapid neural dynamics during cognitive tasks [6]. Conversely, fNIRS offers indirect measurement of neural activity via neurovascular coupling with better spatial specificity and resistance to movement artifacts, making it suitable for naturalistic environments [29] [6]. This complementary relationship enables more comprehensive brain monitoring, as each modality captures different aspects of brain function with different artifact profiles, providing built-in validation through neurovascular coupling principles [29].

Experimental Protocols for Simultaneous EEG-fNIRS Acquisition

Subject Preparation and Equipment Setup

Proper subject preparation is essential for high-quality simultaneous EEG-fNIRS data acquisition. Begin by measuring head circumference and selecting an appropriate integrated EEG-fNIRS cap size (e.g., Model M for 54-58 cm circumferences) [31]. Position the cap according to the international 10-20 or 10-5 system, ensuring comprehensive coverage of regions of interest (typically motor, prefrontal, and parietal cortices depending on the experimental paradigm) [31] [32].

For EEG setup, apply electrolytic gel to achieve electrode impedances below 10 kΩ, using the left or right mastoid as reference (M1/M2) [32] [33]. For fNIRS optode placement, ensure good skin contact without excessive pressure, verifying that source-detector pairs maintain approximately 3 cm separation to achieve optimal cortical sensitivity [31] [32]. Implement a synchronization trigger between EEG and fNIRS systems, typically using event markers transmitted from stimulus presentation software (e.g., E-Prime) to both acquisition systems simultaneously [31].

Table 3: Standardized Equipment Parameters for Simultaneous EEG-fNIRS

Component	EEG Specifications	fNIRS Specifications	Integration Requirements
Acquisition System	g.HIamp amplifier (g.tec) or Neuroscan SynAmps2	NirScan (Danyang Huichuang) or NIRScout (NIRx)	Synchronized triggering capability
Sampling Rate	256-1000 Hz	7.8125-11 Hz	Integer ratio recommended for data fusion
Channel Count	32-64 electrodes	24-90 measurement channels	Co-registered positioning via integrated cap
Reference Scheme	Mastoid (M1/M2) or average reference	Short-separation channels (<1 cm) for superficial artifact removal	Anatomical co-registration of all elements
Stimulus Synchronization	Event markers from E-Prime 3.0 or equivalent	Same event markers as EEG	Simultaneous trigger to both systems

Motor Imagery Paradigm Protocol

Motor imagery (MI) paradigms are widely used in BCI research and provide an excellent framework for demonstrating EEG-fNIRS integration. The following protocol is adapted from established methodologies used in recent multimodal datasets [31] [32]:

• Pre-experiment Preparation: Conduct a grip strength calibration procedure using a dynamometer and stress ball to enhance motor imagery vividness. Have participants perform repeated 5 kg maximal force exertions followed by equivalent force applications using a stress ball, then practice grip training at one contraction per second [31].

• Baseline Recording: Begin with 1-minute eyes-closed resting state followed by 1-minute eyes-open resting state, demarcated by an auditory cue (200 ms beep) [31].

• Trial Structure: Implement the following sequence for each trial:

  • Visual Cue (2 s): Present a directional arrow (left/right) or text/video cue indicating the required MI task [31] [32].
  • Execution Phase (4-10 s): Display a fixation cross following an auditory cue (200 ms beep). Participants perform kinesthetic motor imagery of the specified movement (e.g., hand grasping) at approximately one imagined movement per second [31] [32].
  • Inter-trial Interval (10-15 s): Blank screen or "Rest" instruction appears, allowing hemodynamic responses to return to baseline [31] [32].

• Session Structure: Include at least 2 sessions per subject, with each session containing 15-40 trials per condition. Implement 5-10 minute breaks between sessions to mitigate fatigue effects [31] [32].

• Task Variants: For upper limb MI, include multiple joint movements: hand open/close, wrist flexion/extension, wrist abduction/adduction, elbow pronation/supination, elbow flexion/extension, shoulder pronation/supination, shoulder abduction/adduction, and shoulder flexion/extension [32].

Cognitive Task Protocol (n-back Working Memory)

For cognitive studies, the n-back task provides a well-established paradigm for investigating working memory load:

• Instruction Display (2 s): Present task instructions indicating 0-back, 2-back, or 3-back condition [6].

• Task Period (40 s): Display a random one-digit number every 2 s for 0.5 s, followed by a 1.5-s fixation cross [6]. In 0-back, participants press target button for a specific pre-defined number; in 2-back/3-back, participants indicate whether current number matches the number shown 2/3 trials earlier.

• Rest Period (20 s): Participants focus on a fixation cross while minimizing movement [6].

• Block Structure: Each participant should complete multiple blocks (e.g., 20 trials × 3 series × 3 sessions) to ensure adequate statistical power [6].

Signal Processing Pathways and Data Fusion Approaches

Preprocessing Workflows

Diagram 1: EEG-fNIRS Preprocessing and Fusion Pathway. This workflow illustrates the parallel processing streams for EEG and fNIRS data before multimodal fusion, highlighting key steps including filtering, artifact correction, and feature extraction.

Preprocessing choices significantly impact downstream analysis and decoding performance. For EEG, optimal preprocessing typically includes high-pass filtering (≥1 Hz cutoff) to remove slow drifts, with evidence showing that higher high-pass filter cutoffs consistently increase decoding performance [34]. For fNIRS, motion correction is paramount, with algorithms like CBSI (correlation-based signal improvement), PCA, wavelet-based methods, and spline interpolation being commonly employed [30]. Physiological noise removal via bandpass filtering (0.01-0.5 Hz) effectively isolates hemodynamic responses from cardiac (~1-2 Hz), respiratory (~0.4 Hz), and Mayer wave (~0.1 Hz) interference [30].

Data Fusion Strategies

Diagram 2: Multimodal Fusion Hierarchy. This diagram categorizes the primary fusion strategies for EEG-fNIRS integration, ranging from early data-level fusion to late decision-level fusion, with associated methodologies and applications.

Three primary fusion approaches dominate EEG-fNIRS integration:

• Early Fusion: Concatenating raw data or low-level features before classification, potentially improving performance but requiring temporal alignment of fundamentally different signal dynamics [6].

• Intermediate Fusion: Employing cross-modal attention mechanisms or joint feature learning to model dynamic dependencies between modalities, automatically focusing on relevant signals across time and space [6].

• Late Fusion: Processing modalities independently through specialized architectures (e.g., dual-branch networks) then fusing decisions using methods like Dempster-Shafer Theory (DST) to model and combine uncertainty estimates from each modality [20] [6].

Deep learning approaches have shown particular promise, with architectures like MBC-ATT employing independent branches for each modality with cross-modal attention mechanisms, achieving significant improvements in classification accuracy for cognitive tasks [6]. Similarly, evidence theory-based fusion using Dirichlet distribution parameter estimation and DST has demonstrated 3-5% accuracy improvements in motor imagery classification compared to unimodal approaches [20].

The Scientist's Toolkit: Research Reagent Solutions

Table 4: Essential Research Tools for EEG-fNIRS Acquisition and Analysis

Tool Category	Specific Solutions	Function/Purpose	Implementation Examples
Acquisition Hardware	Integrated EEG-fNIRS caps (Model M)	Simultaneous signal acquisition with co-registered positioning	Custom-designed caps with 32 EEG electrodes, 32 optical sources, 30 photodetectors [31]
Amplification Systems	g.HIamp amplifier (g.tec), Neuroscan SynAmps2	EEG signal amplification and digitization	64-channel systems with 1000 Hz sampling rate, mastoid reference [32]
fNIRS Systems	NirScan (Danyang Huichuang), NIRScout (NIRx)	Optical signal transmission and detection	Continuous wave systems with 690/830 nm wavelengths, ~10 Hz sampling [31] [32]
Stimulus Presentation	E-Prime 3.0, PsychToolbox	Experimental paradigm implementation	Precise timing control with synchronized triggers to both acquisition systems [31]
Preprocessing Tools	MNE-Python, Homer3, EEGLAB	Signal preprocessing and artifact removal	Standardized pipelines for filtering, referencing, motion correction [34] [30]
Fusion Frameworks	MBC-ATT, EEG-fNIRS evidence theory fusion	Multimodal integration and classification	Deep learning with cross-modal attention; Dempster-Shafer theory for uncertainty modeling [20] [6]
Validation Datasets	HEFMI-ICH, TU-Berlin-A, multimodal n-back	Method benchmarking and comparison	Public datasets with simultaneous EEG-fNIRS recordings [20] [31] [6]

The technical and physical basis of EEG and fNIRS signal acquisition provides the foundation for effective multimodal brain imaging research. EEG's millisecond temporal resolution for capturing electrical neural activity complements fNIRS's superior spatial localization of hemodynamic responses, creating a powerful synergistic combination. The experimental protocols and processing pathways detailed in this document establish standardized methodologies for acquiring high-quality simultaneous data. Current research indicates that sophisticated fusion strategies, particularly deep learning approaches with cross-modal attention and evidence theory-based decision fusion, consistently outperform unimodal classification, with improvements of 3-5% in accuracy for motor imagery and cognitive tasks [20] [6]. As the field advances, focus on standardized preprocessing, artifact handling, and open-access datasets will be crucial for accelerating developments in multimodal brain imaging and its applications in BCI, clinical neurology, and cognitive neuroscience.

EEG-fNIRS Fusion Methods: From Basic Concatenation to Advanced Deep Learning

Data-level fusion, often termed early fusion or feature-level fusion, is a data integration strategy wherein raw data or low-level features from multiple sources are combined before being input into a machine learning or statistical model [35] [36]. In the context of cognitive neuroscience and neuroimaging, this approach involves the direct combination of raw or minimally processed signals from different modalities, such as EEG and fNIRS, into a single, unified dataset for subsequent analysis. This method stands in contrast to late fusion (decision-level fusion), where each modality is processed independently and their outputs are combined at the decision stage [35].

The principal advantage of early fusion lies in its potential to model low-level, cross-modal interactions that might be lost when processing modalities separately [36]. For EEG and fNIRS, which capture complementary aspects of brain activity—electrical and hemodynamic, respectively—early fusion allows a model to learn the complex, non-linear relationships between immediate neuronal firing (EEG) and the delayed hemodynamic response (fNIRS) directly from the data. This is particularly valuable for researching brain-computer interfaces, cognitive workload assessment, and the neurovascular correlates of neurological disorders, offering a more holistic view of brain function. However, this approach presents significant challenges, including the need for precise temporal alignment of signals, handling the high-dimensional feature spaces that result from concatenation, and managing the different sampling rates inherent to each modality [35] [37].

Preprocessing Pipelines for Individual Modalities

A rigorous and standardized preprocessing pipeline for each modality is a critical prerequisite for successful data-level fusion. Inconsistent or inadequate preprocessing can introduce artifacts and noise that obscure genuine neural signals and lead to unreliable fusion outcomes.

fNIRS Preprocessing Protocol

The goal of fNIRS preprocessing is to isolate the task-related hemodynamic response function (HRF) by removing physiological, instrumental, and motion artifacts from the raw optical intensity signals [38]. The following protocol, adaptable using toolboxes like HOMER3 or MNE-Python, outlines the essential steps [39].

Protocol: fNIRS Data Preprocessing

• Equipment Setup: Record fNIRS data using a continuous-wave system with sources emitting light at least two wavelengths (e.g., 700 nm and 850 nm) and corresponding detectors. Maintain a source-detector distance typically between 2.5 and 3.5 cm to ensure sufficient cortical penetration [38].
• Channel Selection: Identify and exclude channels where the source-detector distance is too short (e.g., < 1 cm) to detect a neural response [39].
• Intensity to Optical Density: Convert the raw light intensity signals to optical density (OD) to linearize the data with respect to chromophore concentration changes [39].
• Motion Artifact Correction: Apply motion artifact correction algorithms. The Savitzky-Golay filtering method is a common and effective choice for this step [40] [39].
• Bandpass Filtering: Apply a zero-phase bandpass filter (e.g., 0.01 - 0.7 Hz) to the OD data. The high-pass cutoff (0.01 Hz) removes slow drifts, while the low-pass cutoff (0.7 Hz) attenuates high-frequency physiological noise like heart rate (~1 Hz) [39].
• Convert to Hemoglobin: Use the Modified Beer-Lambert Law (MBLL) with an appropriate partial pathlength factor (PPF, often 0.1) to convert the filtered OD data into concentration changes of oxygenated (HbO) and deoxygenated (HbR) hemoglobin [38] [39].
• Epoching: Segment the continuous HbO and HbR data into epochs time-locked to experimental events (e.g., -5 s to +15 s around a stimulus). Apply baseline correction (e.g., from -5 s to 0 s) to each epoch [39].
• Quality Control: Implement the Scalp Coupling Index (SCI) or similar metrics to identify and reject channels with poor signal quality. Manually inspect data and reject epochs with excessive artifact contamination [39].

EEG Preprocessing Protocol

EEG preprocessing aims to isolate neural activity from environmental and physiological artifacts to obtain clean event-related potentials (ERPs) or oscillatory features.

Protocol: EEG Data Preprocessing

• Data Import and Resampling: Import the raw data and downsample to a manageable frequency (e.g., 250 Hz) to reduce computational load, while respecting the anti-aliasing filter [40].
• High-Pass Filtering: Apply a high-pass filter (e.g., 1 Hz) to remove slow drifts and DC offsets [40].
• Line Noise Removal: Use a method like the cleanline function (in EEGLAB) to adaptively remove power line interference (e.g., 50/60 Hz and its harmonics) [40].
• Bad Channel Detection and Interpolation: Automatically detect and remove channels with excessive noise or artifacts. Interpolate the removed channels using spherical splines or other methods [40].
• Re-referencing: Re-reference the data to a common average reference to improve the signal-to-noise ratio.
• Artifact Subspace Reconstruction (ASR): Apply ASR, an automated method for removing short-duration, high-amplitude artifacts, using a parameter value (e.g., 20) that balances artifact removal with signal preservation [40].
• Laplacian Spatial Filter: Optionally, apply a surface Laplacian filter to reduce the effect of volume conduction and focus on cortical sources, which may better correlate with fNIRS signals [40].

Table 1: Summary of Key Preprocessing Steps for fNIRS and EEG

Step	fNIRS	EEG
Primary Goal	Extract HRF	Extract ERP/Oscillations
Filtering	Bandpass (0.01-0.7 Hz)	High-pass (e.g., 1 Hz) & Line Noise Removal
Key Transformation	MBLL (to HbO/HbR)	Re-referencing, Spatial Filtering
Artifact Handling	Savitzky-Golay, SCI	Artifact Subspace Reconstruction (ASR)
Epoching	Around stimulus onset (e.g., -5 to 15 s)	Around stimulus onset (e.g., -0.2 to 1 s)

Data-Level Fusion Methodology

Once the individual modalities are preprocessed, the core process of data-level fusion can begin. This involves bringing the data into a common representational space and combining them.

Temporal Alignment and Synchronization

The first critical step is the precise temporal alignment of the fNIRS and EEG data streams.

• Hardware Synchronization: The ideal approach is to use a shared hardware trigger at the beginning of the experiment and for each experimental event, simultaneously sent to and recorded by both the fNIRS and EEG systems.
• Software Synchronization: If hardware sync is unavailable, software-based alignment using recorded event markers is necessary. This requires accounting for any inherent lags or jitter between the systems.
• Upsampling/Downsampling: Due to the inherently different sampling rates (EEG is typically 250+ Hz, fNIRS is often 10 Hz or lower), one signal must be resampled to match the other. Given the slow nature of the HRF, it is common practice to downsample the EEG data to the fNIRS sampling rate for fusion, or to extract features from the EEG within windows that correspond to fNIRS sample points.

Feature Extraction and Concatenation

After alignment, features are extracted from each modality and combined.

• fNIRS Features: The primary features are the concentration changes of HbO and HbR. These can be used directly from each channel or epoch-averaged.
• EEG Features: For compatibility with the fNIRS HRF, time-domain or frequency-domain features are extracted over sliding windows that correspond to the fNIRS temporal resolution. Common features include:
  • Band Power: The power in key frequency bands (Delta, Theta, Alpha, Beta, Gamma) calculated for each EEG channel.
  • Event-Related Potential (ERP) Amplitude: The mean amplitude of the ERP within specific post-stimulus time windows.
• Concatenation: The extracted feature vectors from fNIRS and EEG are combined into a single, high-dimensional feature vector via concatenation. For example, HbO and HbR values from 16 fNIRS channels and Beta power from 32 EEG channels would create a combined feature vector of (16*2 + 32) = 64 dimensions.

The following diagram illustrates the complete workflow from raw data to a fused feature vector.

Data-Level Fusion Workflow for EEG and fNIRS

Experimental Protocol for a Motor Task Study

This protocol provides a detailed methodology for a classic experiment that generates data suitable for EEG-fNIRS data-level fusion, based on a finger-tapping motor paradigm [39].

Protocol: Simultaneous EEG-fNIRS Recording During Motor Execution

• Objective: To acquire synchronized EEG and fNIRS data during a controlled motor task for investigating the coupling between motor-related cortical potentials/oscillations and hemodynamic responses in the sensorimotor cortex.
• Participants: Recruit healthy, right-handed adult participants. Exclude those with a history of neurological or psychiatric disorders. (Sample size: ~20-30, as justified by a power analysis).
• Stimulus and Task Design:
  • Use an event-related or block-designed paradigm.
  • Condition 1 (Tapping): Participants perform self-paced finger tapping with their right hand for a duration of 5 seconds.
  • Condition 2 (Control): Participants remain at rest for 5 seconds.
  • Present visual cues (e.g., "TAP" or "REST") on a screen to instruct participants. The order of conditions should be randomized.
  • Include an inter-trial interval (jittered, e.g., 15-20 seconds) to allow the hemodynamic response to return to baseline.
• Data Acquisition:
  • fNIRS Setup: Position optodes over the left sensorimotor cortex (C3 position according to the 10-20 system). Use a configuration that covers the primary motor cortex (M1) and supplementary motor area (SMA).
  • EEG Setup: Apply a high-density EEG cap (e.g., 64 channels). Ensure good electrode impedance (< 10 kΩ). Focus recording on electrodes over the sensorimotor cortex (e.g., C3, Cz, CP3).
  • Synchronization: Send a TTL trigger from the stimulus presentation computer to both the fNIRS and EEG amplifiers at the onset of every trial.
• Data Preprocessing: Follow the fNIRS and EEG preprocessing protocols detailed in Section 2 of this document.
• Data-Level Fusion and Analysis:
  • Alignment: Align the EEG and fNIRS data using the shared TTL triggers.
  • Feature Extraction: For each trial, extract the mean HbO concentration from the fNIRS channels over the left M1 from 5-10 seconds post-stimulus. From the EEG, extract the average Beta (13-30 Hz) power decrease (ERD) from electrode C3 in the 1-4 second post-stimulus window.
  • Concatenation: For each trial, create a fused feature vector: [HbO_M1, Beta_ERD_C3].
  • Statistical Modeling: Use the fused feature vector as input to a machine learning model (e.g., Support Vector Machine) to classify Tapping vs. Rest trials, or employ a linear mixed-effects model to assess the relationship between the features and task conditions.

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 2: Essential Equipment and Software for EEG-fNIRS Fusion Research

Item	Function / Application
Continuous-Wave fNIRS System	Records changes in light intensity to calculate hemodynamic activity. Systems with multiple wavelengths (e.g., 760 nm, 850 nm) are required to resolve HbO and HbR [38].
High-Density EEG System	Records electrical activity from the scalp. Systems with 64+ channels and capable of receiving external triggers are recommended for source localization and synchronization.
Trigger Interface Box	A critical hardware component for sending synchronized TTL pulse triggers from the stimulus computer to both EEG and fNIRS systems to ensure precise temporal alignment of data streams.
HOMER3 Software Package	A standard, open-source toolbox (running in MATLAB) specifically designed for fNIRS data preprocessing, visualization, and analysis [40] [39].
EEGLAB Toolbox	A widely used, open-source MATLAB toolbox providing an interactive environment for processing and analyzing EEG data, including advanced techniques like ASR [40].
MNE-Python Library	A powerful, open-source Python library for exploring, visualizing, and analyzing human neurophysiological data, including comprehensive support for both MEG, EEG, and fNIRS [39].

Within the broader scope of data fusion techniques for electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) research, feature-level fusion represents a powerful strategy to overcome the inherent limitations of each modality when used in isolation. This approach involves the extraction and systematic integration of distinctive features from each signal type before they are fed into a classification or analysis model. The fundamental rationale is that EEG and fNIRS provide complementary insights into brain function: EEG captures millisecond-scale electrical activity with high temporal resolution but limited spatial precision, while fNIRS tracks slower hemodynamic changes (changes in oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentrations) with better spatial localization on the cortical surface [41] [6] [9]. By fusing temporal and spectral features from EEG with hemodynamic patterns from fNIRS, researchers can achieve a more comprehensive and robust decoding of brain states, which is critical for applications in cognitive neuroscience, clinical diagnosis, and neurorehabilitation [7] [31].

The subsequent sections detail the core methodology, provide a step-by-step experimental protocol, present validation results, and list essential research tools for implementing this multimodal fusion approach.

Methodological Framework

The proposed feature-level fusion framework involves a structured pipeline from data acquisition to the integrated feature vector ready for classification. The core components are feature extraction from each modality and their subsequent fusion strategy.

Feature Extraction

EEG Feature Extraction: EEG signals are processed to extract features that characterize the brain's electrical activity in time, frequency, and space. Key steps include:

• Preprocessing: Data are bandpass filtered (e.g., 1-40 Hz) and notch-filtered at 50 Hz to remove line noise. Artifacts from eye movements and muscle activity are removed using techniques like Independent Component Analysis (ICA) [41].
• Spectral Decomposition: The clean EEG data are filtered into standard frequency bands: delta (1-3 Hz), theta (3-8 Hz), alpha (8-13 Hz), beta (13-30 Hz), and gamma (30-40 Hz) [41].
• Feature Vector Construction: For each frequency band and channel, features like power spectral density (PSD) are calculated. These features can then be mapped to large-scale brain networks (e.g., the Yeo 7 networks) to create a structured feature vector that summarizes neural activity patterns [41] [42].

fNIRS Feature Extraction: fNIRS signals are processed to yield features related to blood oxygenation changes.

• Preprocessing: The raw light intensity signals are converted into changes in HbO and HbR concentrations. Physiological noise (e.g., cardiac pulsation, respiration) is removed using band-pass filtering [41].
• Hemodynamic Feature Extraction: The mean, slope, and variance of the HbO and HbR time series during task periods are common features. HbO is often prioritized as it is a sensitive indicator of blood oxygen supply and neurovascular coupling activity [41] [43].

Feature Fusion and Integration Strategies

After extraction, features from both modalities are integrated into a single, comprehensive feature vector. The diagram below illustrates the complete workflow.

Beyond simple concatenation, advanced fusion mechanisms can be employed to optimize the interaction between features:

• Cross-Modal Attention: A modality-guided attention mechanism can be used to dynamically weight the importance of EEG and fNIRS features, allowing the model to focus on the most relevant modality for a specific task or time point [6].
• Structured Fusion Networks: Dedicated neural network units (e.g., Spectral-Topological and Spatio-Temporal Feature Fusion Units) can be designed to progressively integrate topological, spectral, and spatiotemporal features, effectively capturing dependencies across different domains [42].

Experimental Protocol for Motor Imagery Decoding

This protocol provides a detailed methodology for a motor imagery (MI) experiment, a common paradigm in brain-computer interface (BCI) research, using synchronous EEG-fNIRS recording and feature-level fusion.

Participant Preparation and Setup

• Participant Screening: Recruit participants based on study criteria (e.g., healthy adults, right-handedness). Exclude individuals with a history of neurological or psychiatric illness, recent concussion, or other contraindications [9] [31]. Obtain informed consent.
• Equipment Setup:
  • Use a hybrid EEG-fNIRS cap where electrodes and optodes are integrated into a single assembly. A typical setup might include a 32-channel EEG configuration and 32 sources/30 detectors for fNIRS, providing ~90 fNIRS measurement channels [31].
  • Ensure fNIRS source-detector separation is approximately 3 cm to achieve adequate cortical penetration [31].
  • Connect the cap to the respective amplifiers and recording systems (e.g., a g.HIamp for EEG and a NirScan system for fNIRS).
• Synchronization: Use a common trigger signal from the experiment presentation software (e.g., E-Prime) to synchronously start both EEG and fNIRS recordings. This ensures temporal alignment of data streams [31].

Data Acquisition Paradigm

• Calibration (Optional but Recommended): To enhance the kinesthetic vividness of motor imagery, have participants perform actual hand movements (e.g., squeezing a stress ball) before the imagery task [31].
• Experimental Run: Implement a cue-based paradigm. A single trial should be structured as follows:
  • Rest Period (Baseline): 15-20 seconds of blank screen or fixation cross.
  • Visual Cue (2 seconds): Display an arrow (e.g., left or right) indicating which hand to imagine moving.
  • Motor Imagery Period (10 seconds): Participants imagine grasping with the designated hand at a rate of one grasp per second while focusing on a central fixation cross. An auditory beep can mark the start.
  • Inter-Trial Interval (15 seconds): Blank screen for rest [31].
• Session Design: Conduct multiple sessions, each containing at least 15 trials per MI task (e.g., left hand vs. right hand). Provide rest breaks between sessions to prevent fatigue.

Data Preprocessing and Analysis

Follow the workflow described in Section 2. The table below summarizes the quantitative performance of various classifiers when applied to fused EEG-fNIRS features, demonstrating the efficacy of this approach.

Table 1: Classifier Performance on Fused EEG-fNIRS Features for State Decoding

Classifier Model	Reported Accuracy	Application Context
Support Vector Machine (SVM)	82.10%	Etomidate use disorder identification [41]
Random Forest (RF)	80.50%	Etomidate use disorder identification [41]
XGBoost	78.40%	Etomidate use disorder identification [41]
Multimodal MBC-ATT (Deep Learning)	Outperformed conventional methods	Cognitive state (n-back, Word Generation) decoding [6]

Validation and Application

The validity of the feature-level fusion approach is confirmed by its ability to identify sensitive neural biomarkers and its superior performance compared to unimodal approaches.

• Identification of Sensitive Biomarkers: Research on individuals with etomidate use disorder revealed significant neurovascular coupling enhancement in the sensorimotor and dorsal attention networks. These features, identifiable only through fused data, showed high sensitivity for classifying the disorder using machine learning classifiers [41].
• Robustness in Clinical Populations: The hybrid approach is particularly valuable for clinical populations like intracerebral hemorrhage (ICH) patients. The HEFMI-ICH dataset provides a resource for developing fusion models that account for unique pathophysiological signatures in these patients, facilitating personalized rehabilitation [31].
• Insights into Shared Neural Networks: Multimodal fusion has been used to elucidate brain activity during motor execution, observation, and imagery. Fused EEG-fNIRS data consistently identified activation in the left inferior parietal lobe and post-central gyrus across all three conditions, highlighting a shared neural region of the Action Observation Network (AON) that was not fully apparent in unimodal analyses [9].

The Scientist's Toolkit

The following table lists essential reagents, materials, and software required to establish a synchronous EEG-fNIRS recording and analysis system.

Table 2: Key Research Reagent Solutions for EEG-fNIRS Feature-Level Fusion

Item Name	Function/Description	Example Specification/Provider
Hybrid EEG-fNIRS Cap	Integrated headgear with electrodes and optodes for simultaneous signal acquisition.	Custom design with 32 EEG electrodes, 32 fNIRS sources, 30 fNIRS detectors [31].
EEG Amplifier	Records electrical brain activity with high temporal resolution.	g.HIamp (g.tec), NeuSen W (Neuracle) [41] [31].
fNIRS System	Measures hemodynamic changes by emitting near-infrared light and detecting its attenuation.	NirSmart-6000A (Danyang Huichuang), NirScan (Danyang Huichuang), Hitachi ETG-4100 [41] [9] [31].
Experiment Control Software	Presents stimuli and sends synchronization triggers to recording devices.	E-Prime (Psychology Software Tools) [31].
Data Analysis Toolkit	Software for preprocessing, feature extraction, and fusion of multimodal data.	EEGLAB (for EEG), custom scripts in MATLAB/Python, Deep Learning frameworks (e.g., PyTorch) [41] [7].
3D Digitizer	Records the precise spatial coordinates of EEG electrodes and fNIRS optodes for accurate brain source localization.	Fastrak (Polhemus) [9].

Decision-level fusion represents a powerful strategy for combining the outputs of multiple classifiers, each potentially processing different data modalities or feature sets, to arrive at a final, more robust decision. Within the context of EEG and fNIRS signal analysis, this approach leverages the complementary strengths of these modalities—EEG provides fine temporal resolution (~5 msec) of electrophysiological activity, while fNIRS offers higher spatial resolution (~1 cm) of hemodynamic responses [4]. Unlike feature-level fusion which concatenates features before classification, decision-level fusion maintains separate processing pathways for each modality until the final decision stage, making it particularly valuable when dealing with inherently different signal characteristics and sampling rates [44]. This protocol document outlines standardized methodologies for implementing decision-level fusion techniques specifically tailored for hybrid EEG-fNIRS brain-computer interfaces and neuro-clinical applications.

Theoretical Foundation and Performance Advantages

Decision-level fusion operates on the principle that combining decisions from multiple classifiers can compensate for individual weaknesses and capitalize on their collective strengths. The mathematical foundation lies in probability theory and statistical inference, where the goal is to optimize the final classification accuracy beyond what any single classifier can achieve. Research has demonstrated that this approach can yield significant improvements in classification performance—one study on mental stress detection reported a +7.76% improvement compared to EEG alone and +10.57% compared to fNIRS alone [44]. Similarly, another implementation for COVID-19 patient health prediction achieved an accuracy of 97.24% by combining random forest, gradient boosting, and extreme gradient boosting classifiers [45].

Table 1: Quantitative Performance Improvements with Decision-Level Fusion

Application Domain	Baseline Accuracy (Best Single Modality)	Decision-Level Fusion Accuracy	Performance Improvement	Citation
Mental Stress Detection (EEG vs fNIRS)	~88% (EEG)	~95.76%	+7.76%	[44]
COVID-19 Health Prediction	94% (Single Model)	97.24%	+3.24%	[45]
Motor Imagery Task Classification	77.6% (Compact hBCI)	96.74%	+19.14%	[44]
Mental Arithmetic Task Classification	90.19% (Feature-level)	98.42%	+8.23%	[44]

The performance advantages stem from several factors: the ability to handle modality-specific characteristics optimally, reduced sensitivity to noise that may affect one modality but not the other, and the capacity to leverage temporal relationships between electrical and hemodynamic responses [4]. Furthermore, decision-level fusion provides a framework for integrating expert knowledge through the careful selection and weighting of individual classifiers, potentially incorporating domain-specific understanding of neurovascular coupling principles.

Experimental Protocols for EEG-fNIRS Decision-Level Fusion

Protocol 1: Ensemble Classifier Fusion with Soft Voting

This protocol outlines the methodology for fusing decisions from multiple heterogeneous classifiers using soft voting, which aggregates probability estimates rather than hard decisions [45].

Step-by-Step Procedure:

• Data Preparation and Preprocessing

  • Acquire simultaneous EEG-fNIRS data during task performance (e.g., motor imagery or mental arithmetic).
  • For EEG: Apply bandpass filtering (0.5-45 Hz), remove artifacts using ICA, and extract spectral features (band powers in theta, alpha, beta, gamma rhythms).
  • For fNIRS: Process raw intensity signals to compute oxygenated (HbO) and deoxygenated (HbR) hemoglobin concentrations using the modified Beer-Lambert law. Extract temporal and morphological features (slope, mean, peak values).

• Individual Classifier Training

  • Train three distinct classifiers on the same training data:
    • Random Forest (RF): Configure with 100 decision trees, Gini impurity criterion.
    • Gradient Boosting (GB): Set learning rate to 0.1, maximum depth to 3, and 100 estimators.
    • Extreme Gradient Boosting (XGB): Use similar parameters as GB but with regularization to reduce overfitting.
  • Employ 10-fold cross-validation for hyperparameter tuning and assess individual classifier performance using accuracy, F1-score, and sensitivity metrics.

• Classifier Calibration

  • Apply Platt scaling or isotonic regression to transform classifier outputs into well-calibrated probability estimates.
  • Verify calibration using reliability diagrams where the expected fraction of positives should match the predicted probability for each bin.

• Decision Fusion Implementation

  • For each test sample, collect the probability estimates from all three calibrated classifiers for each class.
  • Implement soft voting by averaging the probability estimates across classifiers for each class: ( P{fusion}(class=j) = \frac{1}{3} \sum{i=1}^{3} P_i(class=j) )
  • Assign the final class label based on the highest averaged probability: ( \hat{y} = \arg\maxj P{fusion}(class=j) )

• Performance Validation

  • Evaluate the fused classifier on a held-out test set using multiple metrics: accuracy, F1-score, sensitivity, and specificity.
  • Compare performance against individual classifiers and baseline methods using statistical significance testing (e.g., paired t-test).

Protocol 2: Structural Causal Model for Decision Fusion

This protocol adapts the Structural Causal Model (SCM) approach, which explicitly models causal relationships between decisions, for EEG-fNIRS fusion [46].

Step-by-Step Procedure:

• Causal Graph Construction

  • Define three nodes: EEG decision, fNIRS decision, and global decision.
  • Establish directed edges from both preliminary decisions to the global decision node, ensuring an acyclic graph structure.
  • Annotate each node with its possible states (e.g., class labels or probability distributions).

• Structural Equation Specification

  • Define seven fusion objectives based on the diagnostic advantage intervals of each modality. For example:
    • Fusion Objective 1: Favor fNIRS decision when EEG confidence < threshold1 and fNIRS confidence > threshold2.
    • Fusion Objective 2: Favor EEG decision when fNIRS confidence < threshold1 and EEG confidence > threshold2.
    • Fusion Objective 3: Use weighted average when both confidences are moderate.
  • Formalize these objectives as deterministic structural equations: ( GlobalDecision = f(EEGDecision, fNIRSDecision, \theta) ), where ( \theta ) represents parameters learned from training data.

• Model Estimation and Training

  • Use the back-door criterion to identify sufficient sets of variables for estimating causal effects.
  • Estimate structural equation parameters through maximum likelihood estimation or Bayesian methods using training data.
  • Validate the causal structure using conditional independence tests.

• Inference and Decision Making

  • For new test samples, compute preliminary decisions from EEG and fNIRS classifiers.
  • Apply the structural equations with estimated parameters to combine these decisions into a global decision.
  • Account for uncertainty in causal estimates through probabilistic inference.

Table 2: Research Reagent Solutions for EEG-fNIRS Decision-Level Fusion

Reagent Category	Specific Tool/Algorithm	Function in Experimental Protocol
Base Classifiers	Random Forest, Gradient Boosting, XGBoost, SVM, k-NN	Generate preliminary decisions from features; provide diversity for ensemble [44] [47] [45]
Fusion Methods	Soft Voting, Structural Causal Models, Bayesian Networks, Neural Networks	Combine preliminary decisions into final output using specific mathematical frameworks [46] [45]
Feature Extraction	Fractal Dimension, Higher-Order Spectra, Entropy Measures, Band Power	Transform raw signals into informative features for classification [47]
Feature Selection	Genetic Algorithms, Mutual Information, Atomic Search Optimization	Identify most relevant features to reduce dimensionality and improve performance [4] [47]
Signal Processing	ICA, Bandpass Filters, Motion Artifact Removal Algorithms	Preprocess raw data to enhance signal quality and remove noise [4] [47]
Validation Metrics	Accuracy, F1-Score, Sensitivity, Specificity, Statistical T-Tests	Quantify and compare performance across different fusion strategies [44] [45]

Implementation Considerations for Neuroclinical Applications

Successful implementation of decision-level fusion in EEG-fNIRS research requires careful attention to several practical considerations. First, temporal synchronization between EEG and fNIRS systems is crucial, as the hemodynamic response measured by fNIRS typically lags behind the electrical activity captured by EEG by several seconds [4]. Implementation should incorporate appropriate temporal alignment techniques, such as cross-correlation analysis or the use of shared trigger signals. Second, classifier diversity is essential for effective fusion—selecting classifiers with different inductive biases (e.g., tree-based, probabilistic, and margin-based models) creates complementarity that fusion can leverage [45]. Third, computational efficiency must be balanced with performance requirements, particularly for potential real-time applications like brain-computer interfaces where decision latency matters [44].

When deploying these methods in clinical or pharmaceutical research settings, particular attention should be paid to model interpretability. While complex ensemble methods may offer superior performance, the ability to understand and explain decisions is often critical for diagnostic applications. Techniques such as SHAP (SHapley Additive exPlanations) values or LIME (Local Interpretable Model-agnostic Explanations) can be incorporated to maintain this interpretability while benefiting from the performance advantages of decision-level fusion. Additionally, researchers should implement comprehensive validation protocols that include not only standard performance metrics but also assessments of robustness across patient populations, stability over multiple sessions, and generalizability to new data distributions.

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a paradigm shift in non-invasive brain imaging, leveraging the complementary strengths of each modality to overcome their individual limitations. EEG provides millisecond-level temporal resolution crucial for capturing fast neural dynamics but suffers from poor spatial resolution and sensitivity to artifacts [6] [48]. In contrast, fNIRS measures hemodynamic responses with better spatial localization and robustness to motion artifacts but has limited temporal resolution [29]. This complementary relationship has motivated the development of advanced data fusion techniques, particularly cross-modal attention mechanisms and multi-branch neural networks, which enable more sophisticated integration of electrophysiological and hemodynamic brain activity patterns for enhanced brain-computer interface (BCI) performance and cognitive state decoding [6] [49] [48].

Core Architectural Frameworks

Multi-Branch Neural Network Architectures

Multi-branch architectures have emerged as a dominant framework for multimodal brain signal processing, allowing specialized pathway design for modality-specific feature extraction.

MBC-ATT (Multi-Branch Convolutional Neural Network with Attention) employs independent branch structures to process EEG and fNIRS signals separately, thereby leveraging the unique advantages of each modality. The architecture implements a late fusion strategy with a cross-modal attention mechanism that selectively emphasizes relevant features across modalities, strengthening the model's ability to focus on task-relevant signals [6] [48]. Experimental results on n-back and word generation datasets demonstrate that this approach outperforms conventional methods in classification performance, validating its effectiveness for brain-computer interfaces [50] [48].

Multi-Branch GAT-GRU-Transformer utilizes three parallel branches for comprehensive feature extraction: a Graph Attention Network (GAT) for modeling spatial relationships among EEG channels, a hybrid Gated Recurrent Unit (GRU) and Transformer module for capturing temporal dependencies, and one-dimensional Convolutional Neural Networks (1D CNN) for extracting frequency-specific information [49]. This architecture achieved a classification accuracy of 55.76% on five-class motor imagery tasks, significantly outperforming single-dimensional approaches [49].

Cross-Modal Attention Mechanisms

Cross-modal attention mechanisms represent a significant advancement beyond simple feature concatenation or decision-level fusion. These mechanisms dynamically model dependencies between modalities, allowing the model to automatically focus on the most relevant modalities and brain region signals according to specific task states [6] [48].

The attention mechanism in MBC-ATT implements a modality-guided attention strategy that selectively integrates information through joint modeling of cross-modal features. This approach addresses the limitations of static fusion strategies by dynamically adjusting the contribution of each modality, thereby enhancing the synergistic capability of fused signals and improving decoding accuracy and robustness [48]. This dynamic weighting is particularly valuable in complex cognitive tasks where the relative importance of EEG and fNIRS signals may vary throughout task execution.

Table 1: Performance Comparison of Multi-Branch Architectures

Architecture	Modality	Task	Performance	Key Innovation
MBC-ATT [6] [48]	EEG-fNIRS	n-back & Word Generation	Outperformed conventional approaches	Cross-modal attention fusion
Multi-Branch GAT-GRU-Transformer [49]	EEG	Five-class Motor Imagery	55.76% accuracy	Spatial, temporal, frequency feature integration
Mutual Information-Based Fusion [4]	EEG-fNIRS	Visuo-mental Task	Considerably improved hybrid classification	Mutual information feature selection

Experimental Protocols and Methodologies

Dataset Specifications and Paradigms

Multiple standardized datasets have been developed to validate multimodal fusion architectures, each with specific experimental paradigms tailored to different research questions.

n-back Working Memory Task: Participants complete 0-back, 2-back, and 3-back conditions in randomized blocks. Each block consists of a 2-second instruction display, followed by a 40-second task period where a random one-digit number is displayed every 2 seconds for 0.5 seconds, followed by a 1.5-second fixation cross. After the task period, participants enter a 20-second rest period focusing on a fixed cross [6] [48]. Each participant typically performs 180 trials (20 trials × 3 series × 3 sessions) to ensure data adequacy and reliability.

Word Generation Task: This paradigm investigates neural activity characteristics of language-related brain regions. Each trial consists of a 2-second task prompt, a 10-second task execution period, and a 13-15 second rest period. During the task period, a random letter is displayed, and participants must quickly generate and silently list as many words as possible starting with that letter while avoiding repetition [48].

Motor Imagery Paradigms: Standardized motor imagery experiments typically involve visual cue presentation (2 seconds) followed by an execution phase (4-10 seconds) where participants perform kinesthetic motor imagery of specific movements, followed by rest periods (10-20 seconds) [31] [32]. The HEFMI-ICH dataset incorporates a unique grip strength calibration procedure using a dynamometer and stress ball to enhance motor imagery vividness and consistency [31].

Data Acquisition Parameters

Table 2: Standardized Data Acquisition Parameters

Parameter	EEG Specifications	fNIRS Specifications
Sampling Rate	256-1000 Hz [31] [32]	7.8125-11 Hz [31] [32]
Electrode/Optode Configuration	32-64 electrodes following international 10-20 system [31] [32]	24-90 channels with 3cm source-detector separation [31] [32]
Key Metrics	Bandpower, amplitude, phase, frequency features [4]	Oxygenated (HbO) and deoxygenated hemoglobin (HbR) concentrations [4]
Reference Scheme	Left mastoid (M1) reference [32]	Short-separation channels for systemic artifact removal [29]
Filter Settings	0.5-100 Hz bandpass, 50 Hz notch filter [32]	Bandpass filter to isolate hemodynamic responses (typically 0.01-0.2 Hz) [29]

Signal Preprocessing Workflows

EEG preprocessing typically involves bandpass filtering (0.5-100 Hz), notch filtering at 50 Hz for power line interference removal, and artifact removal techniques including independent component analysis (ICA) for ocular and muscle artifact rejection [29] [32]. fNIRS processing includes converting raw light intensities to optical densities, motion artifact detection and correction, bandpass filtering to isolate hemodynamic responses, and conversion to hemoglobin concentration changes using the Modified Beer-Lambert Law [29]. Temporal synchronization between modalities is crucial and is typically achieved using event markers transmitted from experimental presentation software such as E-Prime 3.0 [31].

Implementation Framework

Architectural Workflow

The following diagram illustrates the complete workflow for multimodal EEG-fNIRS processing using cross-modal attention mechanisms:

Cross-Modal Attention Mechanism

The cross-modal attention mechanism implements a sophisticated feature weighting system as shown in the following diagram:

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Research Tools for EEG-fNIRS Fusion Studies

Tool Category	Specific Examples	Function & Application
Acquisition Systems	g.HIamp amplifier (EEG) [31], NirScan fNIRS system [31], NIRScout system [32]	Synchronized multimodal data acquisition with precise temporal alignment
Experimental Paradigm Software	E-Prime 3.0 [31], PsychoPy, Presentation	Precise stimulus presentation and event marker generation for temporal synchronization
Signal Processing Tools	MATLAB with EEGLAB [29], NIRS-KIT [29], MNE-Python, Homer2	Preprocessing, artifact removal, and feature extraction pipelines
Hybrid Caps	Custom EEG-fNIRS caps with 32 electrodes + 32 sources + 30 detectors [31]	Co-localized electrophysiological and hemodynamic measurement from same cortical regions
Deep Learning Frameworks	PyTorch, TensorFlow, Braindecode	Implementation of multi-branch architectures and attention mechanisms
Validation Metrics	Classification accuracy, mutual information [4], SHAP analysis [49]	Performance quantification and model interpretability

Applications and Validation

The application of these advanced architectures spans multiple domains, demonstrating significant performance improvements over unimodal approaches. In cognitive state decoding, the MBC-ATT framework has shown superior performance in classifying working memory load during n-back tasks and language processing during word generation tasks [6] [48]. For motor imagery classification, multi-branch architectures have achieved accuracy improvements of 5-10% compared to unimodal systems, with particularly promising applications in neurorehabilitation for intracerebral hemorrhage patients [31].

Mutual information-based feature selection approaches have demonstrated considerable improvement in hybrid classification performance compared to individual modalities and conventional classification without feature selection [4]. This approach optimizes complementarity, redundancy, and relevance between multimodal features with respect to class labels, suggesting potential efficacy for wider neuro-clinical applications.

Validation methodologies typically include k-fold cross-validation, subject-independent testing to assess generalizability, and ablation studies to quantify the contribution of individual architectural components [49]. Additionally, interpretability techniques such as SHAP analysis and Phase Locking Value calculations help identify critical EEG channels and frequency bands, aligning model decisions with known neurophysiological principles [49].

Cross-modal attention mechanisms and multi-branch neural networks represent the cutting edge of EEG-fNIRS data fusion, offering sophisticated solutions that leverage the complementary nature of electrophysiological and hemodynamic brain signals. These architectures enable dynamic, task-adaptive integration of multimodal information, moving beyond the limitations of static fusion strategies. As these approaches continue to evolve, they hold significant promise for advancing BCI systems, cognitive state monitoring, and clinical applications in neurological rehabilitation and drug development research. The standardized protocols and architectural frameworks presented in this document provide a foundation for implementing these advanced data fusion techniques in both research and clinical settings.

Application Notes: Multimodal EEG-fNIRS in Clinical Practice

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a transformative approach in neuroimaging, leveraging the complementary strengths of each modality. EEG provides millisecond-level temporal resolution for capturing rapid neural electrical activity, while fNIRS offers superior spatial localization for tracking hemodynamic responses through cortical blood oxygen changes [6]. This synergistic combination enables more comprehensive brain decoding with high spatiotemporal resolution, particularly valuable for real-world clinical applications [7].

Data fusion techniques for EEG and fNIRS can be implemented at multiple levels, including early fusion (combining raw data or low-level features), feature-level fusion (integrating extracted features before classification), and late fusion (combining decisions or high-level features) [6]. Recent advances incorporate cross-modal attention mechanisms and deep learning architectures that dynamically weight the contribution of each modality based on task demands, significantly enhancing classification accuracy and robustness across various clinical applications [6].

Key Advantages of EEG-fNIRS Integration

Table 1: Comparative Analysis of Neuroimaging Modalities

Modality	Temporal Resolution	Spatial Resolution	Artifact Resistance	Portability	Primary Applications
EEG	Millisecond-level	Limited (cm)	Low	High	Seizure detection, consciousness assessment, basic BCI
fNIRS	Seconds	Good (1-2 cm)	Moderate	High	Functional localization, cognitive monitoring
EEG-fNIRS Fusion	High (via EEG)	Good (via fNIRS)	Moderate-High	High	Advanced BCI, drug craving detection, disorder diagnosis
fMRI	Seconds	Excellent (mm)	High	Low	Gold standard for spatial localization

Experimental Protocols

Protocol 1: Drug Addiction Severity Assessment

Objective: To objectively classify individuals with substance addiction versus healthy controls using bimodal EEG-fNIRS data and deep learning.

Experimental Design: The protocol employs a visual trigger paradigm to elicit drug cravings in individuals with substance addiction, with concurrent EEG and fNIRS monitoring [51].

Participant Profile:

• Target Population: 36 individuals with documented drug addiction and 20 healthy controls
• Stimuli: 56 drug-related images per participant
• Setting: Controlled laboratory environment

Data Acquisition Specifications:

• EEG Configuration: 52 electrodes covering frontal, parietal, occipital, and temporal regions
• fNIRS Configuration: 21 channels focused on frontal areas directly implicated in addiction responses
• Synchronization: Simultaneous recording with timestamp alignment

Processing Workflow:

• Preprocessing: Artifact removal, filtering, and signal normalization
• Feature Extraction: Temporal patterns from EEG, hemodynamic responses from fNIRS
• Model Architecture: AR-TSNET deep learning network with Tception (EEG) and Sception (fNIRS) modules
• Classification: Feature-level fusion with attention mechanisms and residual connections
• Validation: Sixfold cross-validation with strict separation of training and testing sets

Performance Metrics: The protocol achieves 92.6% classification accuracy with an AUC of 0.903, significantly outperforming single-modal approaches [51].

Protocol 2: Stroke Neurorehabilitation with BCI

Objective: To enhance motor recovery in stroke patients through BCI-based rehabilitation integrating motor imagery and motor attempts.

Experimental Design: Randomized double-blind controlled clinical trial with 48 ischemic stroke patients (25 BCI group, 23 control group) undergoing 20-minute upper and lower limb training sessions for two weeks [52].

Intervention Protocol:

• BCI Group: Real-time EEG-based feedback system with motor intention decoding
• Control Group: Identical setup without real-time feedback (sham intervention)
• Task Paradigm: Integration of motor imagery and motor attempt tasks
• Feedback Mechanism: Virtual reality training module and rehabilitation robot

Assessment Framework:

• Primary Outcome: Fugl-Meyer Assessment for Upper Extremity (FMA-UE)
• Secondary Outcomes: EEG metrics (DAR, DABR), EMG muscle activity, fNIRS-based functional connectivity
• Timeline: Pre-intervention, post-intervention, and follow-up assessments

Key Findings: The BCI group demonstrated significantly greater improvement in upper extremity motor function (ΔFMA-UE: 4.0 vs. 2.0, p = 0.046) with enhanced functional connectivity in motor-related brain regions [52].

Protocol 3: Cognitive Motor Dissociation Detection

Objective: To identify patients with cognitive motor dissociation (CMD) among disorders of consciousness (DOC) populations using fNIRS with motor imagery tasks.

Experimental Design: fNIRS measurement of hemodynamic responses in 70 prolonged DOC patients during command-driven hand-open-close motor imagery tasks [53].

Participant Profile:

• Target Population: 30 vegetative state/unresponsive wakefulness syndrome (VS/UWS), 20 minimally conscious state minus (MCS-), and 20 minimally conscious state plus (MCS+) patients
• Control Group: 70 healthy individuals
• Inclusion Criteria: Right-handed, DOC duration >28 days, complete auditory brainstem evoked potentials

Task Paradigm:

• Motor Imagery Task: 20-second hand-open-close imagery followed by 20-second rest
• Block Design: 5 repetitions with 50-second pre- and post-baseline periods
• Total Duration: 300 seconds per session
• Stimulus Presentation: Auditory commands via verbal instruction

Analysis Approach:

• Feature Extraction: 7 hemodynamic response features during task and rest conditions
• Classification: Support vector machine with genetic algorithm optimization
• Outcome Validation: 6-month follow-up using Glasgow Outcome Scale-Extended (GOSE)

Key Findings: Identification of 7 CMD patients (4 VS/UWS, 3 MCS-) with significantly better outcomes (3/4 vs. 1/31, P = 0.014) compared to non-CMD patients [53].

Signaling Pathways and Neurovascular Coupling

The effectiveness of EEG-fNIRS integration relies fundamentally on the principle of neurovascular coupling - the tightly regulated relationship between neural activity and subsequent hemodynamic responses.

Temporal Relationship: The EEG signal detection precedes the fNIRS hemodynamic response by approximately 1-5 seconds, creating a complementary temporal signature that can be leveraged for more accurate brain state classification [7] [54].

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Research Materials and Technologies for EEG-fNIRS Research

Category	Specific Solutions	Technical Specifications	Primary Function	Example Applications
EEG Systems	8-64 channel active electrode systems (e.g., g.tec LadyBird)	Sampling rate: 256-512 Hz, Impedance: <10 kΩ	Capture millisecond-level electrical brain activity	Motor intention decoding, cognitive state assessment [52] [55]
fNIRS Systems	Continuous-wave systems (e.g., NirScan-6000A)	Wavelengths: 703-850 nm, Sources: 24, Detectors: 24	Measure hemodynamic responses via HbO/HbR concentration changes	Localization of neural activation, consciousness assessment [53]
Multimodal Caps	Integrated EEG-fNIRS headcaps	Predefined optode/electrode layouts per 10-20 system	Ensure proper spatial co-registration of modalities	Simultaneous bimodal data acquisition [51]
Stimulation Paradigms	Visual trigger systems, Auditory command interfaces	Precision timing, Counterbalanced design	Elicit specific neural responses for experimental control	Drug cue reactivity, motor imagery tasks [51] [53]
Analysis Platforms	MATLAB, Python with MNE, Homer2, BCILab	Custom scripts for fusion algorithms	Implement data fusion pipelines and classification	Feature extraction, cross-modal attention fusion [51] [6]
Validation Tools	Clinical assessment scales (FMA-UE, CRS-R)	Standardized scoring protocols	Ground truth measurement for algorithm validation	Outcome verification, clinical correlation [52] [53]

Performance Metrics and Clinical Validation

Table 3: Quantitative Performance Comparison Across Applications

Application Domain	Classification Accuracy	Superiority Over Single Modality	Key Performance Metrics	Clinical Validation
Drug Addiction Detection	92.6% [51]	Significant improvement (p<0.05)	AUC: 0.903, F1-Score: 0.91	56 samples/participant, n=56 [51]
Stroke Rehabilitation	ΔFMA-UE: 4.0 vs. 2.0 (p=0.046) [52]	Clinical significance demonstrated	DAR decrease (p=0.031), DABR decrease (p<0.001)	RCT, n=48, double-blind [52]
Cognitive State Decoding	Superior to conventional approaches [6]	Enhanced cross-modal synergy	Improved robustness to artifacts	n-back and WG tasks, n=26 [6]
CMD Identification	Favorable outcome: 3/4 vs 1/31 (p=0.014) [53]	Identified 7 CMD from 70 DOC	6-month GOSE follow-up	Diagnostic study, n=70 patients + 70 controls [53]

Implementation Considerations

Technical Integration Challenges:

• Synchronization: Hardware-level timing synchronization critical for precise multimodal data alignment
• Artifact Handling: fNIRS more robust to motion artifacts than EEG, requiring modality-specific correction approaches [6]
• Optode Placement: Strategic positioning to maximize coverage of target brain regions while minimizing interference

Clinical Deployment Factors:

• Portability: Both EEG and fNIRS offer advantages for bedside monitoring compared to fMRI [53]
• Patient Tolerance: Non-invasive nature supports repeated measurements and long-term monitoring
• Analysis Complexity: Deep learning approaches require substantial computational resources and expertise

The integration of EEG and fNIRS through advanced data fusion techniques represents a paradigm shift in clinical neuroscience, enabling unprecedented insights into brain function across neurological and psychiatric conditions. These approaches show particular promise for conditions where traditional assessment methods face limitations, such as disorders of consciousness, addiction, and stroke recovery.

This application note details a pioneering study that successfully classified individuals with drug addiction from healthy controls with an accuracy of 92.6% [56] [57]. This work is situated within a broader research thesis focused on advancing data fusion techniques for Electroencephalogram (EEG) and functional Near-Infrared Spectroscopy (fNIRS) signals. The objective was to overcome the limitations of traditional, subjective assessment methods—such as psychological scales and self-reports—by developing an objective, physiological data-driven model [56]. The core innovation lies in the AR-TSNET deep learning algorithm, which performs feature-level fusion of EEG and fNIRS data, leveraging the complementary strengths of these two modalities to achieve superior classification performance [56] [57].

Experimental Protocols and Workflow

The experiment followed a structured protocol designed to elicit, capture, and analyze brain activity related to drug addiction.

Participant Recruitment and Dataset Composition

The study compiled a dataset from 56 participants [56] [57]. Adherence to ethical guidelines was confirmed, including institutional ethical approval and obtaining informed consent from all participants [57].

• Drug Addiction Group: 36 individuals undergoing drug rehabilitation.
• Healthy Control Group: 20 healthy individuals.

Each participant contributed 56 samples, resulting in a total dataset corresponding to responses to drug-related image stimuli [56].

Data Acquisition and Stimulus Paradigm

A visual trigger paradigm was employed to induce drug cravings in the addicted individuals [56] [57]. During this task, simultaneous EEG and fNIRS signals were recorded.

• EEG Acquisition: Brain electrical activity was recorded from 52 electrodes placed across the frontal, parietal, occipital, and temporal regions [56].
• fNIRS Acquisition: Hemodynamic activity was measured from 21 channels located in the frontal area, a region directly implicated in the neurobiology of addiction [56].

Data Processing and Analysis with AR-TSNET

The acquired data were processed and classified using the novel AR-TSNET deep learning network. The following workflow outlines the key stages of the experimental protocol.

The AR-TSNET Architecture Protocol

The AR-TSNET model was designed with specialized sub-networks for each modality, followed by fusion and classification.

• Modality-Specific Feature Extraction:

  • The Tception module was used to process the EEG data, leveraging its design to capture temporal patterns effectively [56] [57].
  • The Sception module was dedicated to extracting relevant features from the fNIRS data [56] [57].

• Feature-Level Fusion: The features extracted by Tception and Sception were combined at a feature level, creating a unified, information-rich representation for the classifier [56].

• Attention Mechanism and Residual Connections:

  • An attention mechanism was incorporated to dynamically assign weights to the features, reducing the interference of redundant information and allowing the model to focus on the most salient biomarkers [56] [57].
  • Residual connections were used throughout the network to mitigate the problem of information loss as network depth increased, thereby enhancing model stability and robustness [56].

• Model Training and Validation:

  • The model was trained for 400 epochs [56].
  • Performance was evaluated using sixfold cross-validation, ensuring a robust estimate of the model's generalizability [56].
  • Techniques including dropout, L2 regularization, and early stopping were employed to prevent overfitting [56].

Performance Metrics and Comparative Analysis

The model's performance was rigorously quantified using standard metrics. A comparison with single-modality approaches demonstrates the clear advantage of multimodal fusion.

The model achieved an average classification accuracy of 92.6% with a standard deviation of 5.56 across the six folds [56]. The area under the Receiver Operating Characteristic (ROC) curve (AUC) was 0.903, confirming the model's strong ability to discriminate between the two groups [56].

Table 1: Overall Performance Metrics of the AR-TSNET Model

Metric	Value	Interpretation
Accuracy	92.6%	Overall proportion of correct predictions [56]
AUC	0.903	Excellent model discriminative ability [56]
Standard Deviation	5.56	Variance in accuracy across cross-validation folds [56]

Single-Modality vs. Multimodal Fusion

A critical analysis was conducted to compare the performance of the fused model against models using only EEG or only fNIRS data. The results conclusively show that the bimodal approach provides superior performance across all evaluated metrics [56].

Table 2: Comparison of Single-Modal vs. Bimodal Fusion Performance

Model Type	Precision	Recall	F1-Score
EEG Only	< 90%	< 90%	< 90%
fNIRS Only	< 90%	< 90%	< 90%
EEG-fNIRS Fusion (AR-TSNET)	> 90%	> 90%	> 90%

Note: Exact values for single-modal results were not fully detailed in the source, but were reported to be below the performance of the fused model, which exceeded 90% on all metrics [56].

The Scientist's Toolkit: Research Reagent Solutions

This section outlines the essential materials, software, and analytical tools used in this study, providing a resource for researchers seeking to replicate or build upon this work.

Table 3: Key Research Reagents and Tools for EEG-fNIRS Fusion Studies

Item / Technique	Function / Description	Role in the Present Study
EEG System	Records electrical activity of the brain with high temporal resolution [6].	Acquired millisecond-level neural potential fluctuations from 52 scalp electrodes [56].
fNIRS System	Measures hemodynamic changes (oxy-/deoxy-hemoglobin) with good spatial resolution [6].	Captured blood oxygen changes in 21 frontal cortex channels related to craving [56].
Visual Trigger Paradigm	A set of standardized stimuli (e.g., images) designed to provoke a specific cognitive or emotional state.	Used drug-related images to reliably elicit craving responses in the addicted group [56] [57].
AR-TSNET Algorithm	A custom deep learning network for feature-level EEG-fNIRS fusion.	Core classification model featuring Tception, Sception, attention, and residual connections [56] [57].
Attention Mechanism	A deep learning component that dynamically weights the importance of input features.	Enhanced model focus on the most critical EEG and fNIRS features for classification [56].
k-Fold Cross-Validation	A resampling procedure used to evaluate a model on limited data samples.	Employed sixfold cross-validation to ensure robust and generalizable performance metrics [56].

This case study demonstrates that deep learning-based fusion of EEG and fNIRS signals is a highly effective and promising objective method for assessing drug addiction severity [56] [57]. The achieved accuracy of 92.6% underscores the transformative potential of multimodal data fusion in neuroengineering and clinical diagnostics.

Within the broader thesis on data fusion techniques for EEG and fNIRS, this work provides a concrete implementation and validation of a feature-level fusion strategy. It highlights how overcoming the inherent limitations of each single modality—by combining the high temporal resolution of EEG with the better spatial resolution and artifact resistance of fNIRS—can yield a more comprehensive picture of brain activity and significantly enhance classification accuracy in real-world clinical applications, such as addiction medicine [7] [6] [58]. The success of the AR-TSNET model paves the way for its application in other neuropsychiatric disorders where objective biomarkers are urgently needed.

Overcoming Practical Challenges in Multimodal Signal Acquisition and Processing

The fusion of Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) provides a powerful neuroimaging approach with high spatiotemporal resolution, but its signal quality is fundamentally limited by various artifacts. Motion, physiological, and environmental noises can significantly obscure genuine neural signatures, leading to inaccurate data interpretation and conclusions. Effective artifact management is therefore not merely a preprocessing step but a critical foundation for reliable data fusion, particularly in studies involving drug development where subtle neurophysiological changes are of paramount interest. This document outlines standardized protocols for identifying and removing these contaminations to ensure data integrity in multimodal research.

Classification and Characteristics of Artifacts

Artifacts in combined EEG-fNIRS studies can be categorized based on their origin. The table below summarizes the primary types, their characteristics, and common removal approaches.

Table 1: Classification and Overview of Artifacts in EEG-fNIRS Signals

Artifact Category	Specific Type	Typical Frequency Range	Main Causes	Common Removal Approaches
Motion Artifacts	Head Movement	Slow drifts to sharp spikes (< 0.1 Hz to >10 Hz)	Head displacement, optode movement [59] [60]	Accelerometer-based correction [59], Spline interpolation [59] [61], WPD-CCA [62]
	Jaw/Facial Movement	Variable	Talking, chewing [59]	Computer vision tracking [60], Channel rejection
	Gait-Related	Periodic (~1-2 Hz)	Heel strike during walking [63]	Adaptive filtering, Deep Learning (e.g., Motion-Net) [63]
Physiological Noise	Cardiac	~1 Hz (0.8-1.5 Hz)	Heartbeat/pulse [64] [61]	Band-pass filtering, Adaptive filtering [61], WPD [62]
	Respiratory	~0.2-0.3 Hz (0.1-0.5 Hz)	Breathing [64] [61]	Band-pass filtering, MODWT [65]
	Mayer Waves	~0.1 Hz	Very-low-frequency blood pressure oscillations [64] [61]	Band-pass filtering, MODWT-LSTM prediction [65]
	Systemic Hemodynamic	< 0.1 Hz	Global blood flow changes in scalp [66] [64]	Short-Separation Regression [66] [64], PCA/GLM [66]
Environmental Noise	Powerline Interference	50/60 Hz	AC power lines	Notch filtering
	Technical Artifacts	Broad Spectrum	Equipment limitations, cable movement [63]	Proper grounding, Hardware inspection

Experimental Protocols for Artifact Identification and Removal

Protocol 1: Comprehensive Data Acquisition for Artifact Handling

Objective: To acquire simultaneous EEG-fNIRS data with auxiliary information necessary for robust artifact identification and correction.

Materials:

• A synchronized EEG-fNIRS system.
• Short-separation fNIRS channels (source-detector distance ~0.5-1.0 cm) [66] [64].
• Auxiliary hardware: Tri-axial accelerometers attached to the head [59], pulse oximeter, respiratory belt [66].
• Video recording system (optional, for computer vision analysis) [60].

Procedure:

• Setup: Apply the EEG cap and fNIRS optodes according to the international 10-20 system. Ensure short-separation fNIRS detectors are interspersed among long-separation channels.
• Auxiliary Sensor Attachment: Securely attach accelerometers to the cap to monitor head motion. Connect the pulse oximeter and respiratory belt.
• Synchronization: Initiate all systems (EEG, fNIRS, auxiliary sensors) and establish a common synchronization pulse.
• Baseline Recording: Record a 5-10 minute resting-state baseline.
• Task Recording: Proceed with the experimental paradigm.
• Motion Tasks: At the end of the session, instruct the participant to perform a series of standardized head movements (e.g., nodding, shaking) to capture motion artifact profiles for later validation [60].

Diagram 1: Artifact-Resilient Data Acquisition Workflow

Protocol 2: A Two-Stage Pipeline for Motion Artifact Removal

Objective: To systematically remove motion-induced artifacts from fNIRS and EEG signals.

Background: Motion artifacts (MAs) are a dominant source of noise, characterized by sudden, high-amplitude spikes or shifts in the signal baseline [59] [60]. The following pipeline leverages a powerful two-stage algorithmic approach.

Procedure for fNIRS Signal Correction:

• Identification: Identify motion-contaminated segments in the fNIRS signal using a moving standard deviation threshold or accelerometer data [59].
• Correction via WPD-CCA: a. Wavelet Packet Decomposition (WPD): Decompose the contaminated signal into multiple frequency sub-bands using a Daubechies (db1) or Fejer-Korovkin (fk4) wavelet packet [62]. b. Artifact Reconstruction: Reconstruct the artifact signal from the sub-bands containing the motion-related components. c. Canonical Correlation Analysis (CCA): Apply CCA between the original signal and the reconstructed artifact to further separate and remove the motion components. This two-stage method (WPD-CCA) has been shown to achieve an average signal-to-noise ratio (SNR) improvement of 16.55 dB and artifact reduction (η) of 41.40% for fNIRS signals [62].

Procedure for EEG Signal Correction (Deep Learning Approach):

• Data Preparation: For subject-specific correction, use segments of motion-corrupted EEG and their corresponding clean segments (or accelerometer data as a noise reference) [63].
• Model Training: Train a convolutional neural network (CNN) like Motion-Net based on a U-Net architecture. The model learns the mapping from noisy EEG inputs to clean outputs.
• Artifact Removal: Process the motion-contaminated EEG signals through the trained Motion-Net model. This approach has demonstrated an average artifact reduction of 86% and an SNR improvement of 20 dB [63].

Diagram 2: Motion Artifact Removal Pipeline

Protocol 3: Advanced Physiological Noise Filtering in fNIRS

Objective: To filter out systemic physiological noises that overlap with the task-evoked hemodynamic response.

Background: Physiological noises (cardiac, respiratory, Mayer waves) reside in frequency bands close to the hemodynamic response (~0.01-0.1 Hz). Simple band-pass filtering is often insufficient, especially for global systemic components captured in the superficial layers [64] [65]. This protocol uses a model-based approach.

Materials:

• fNIRS data with short-separation (SS) channels.
• Auxiliary physiological recordings (heart rate, respiration).

Procedure:

• Model Definition: Model the measured fNIRS signal y(t) in a long-separation channel as a linear combination of components [61]: y(t) = β * HRF(t) + α * SS(t) + Σ [b_m * sin(2πf_m t)] + ε(t) where:
  • HRF(t) is the expected hemodynamic response function.
  • SS(t) is the signal from a short-separation channel.
  • Σ [b_m * sin(2πf_m t)] represents the sum of sinusoidal models for cardiac, respiratory, and Mayer wave frequencies.
  • ε(t) is the residual error.
• Parameter Estimation: Use a Recursive Least-Squares Estimator (RLSE) with a forgetting factor to adaptively estimate the unknown parameters (β, α, b_m) [61]. This method is effective for both offline and online processing.
• Noise Subtraction: The corrected hemodynamic response is obtained by subtracting the estimated noise components (scaled SS signal and physiological sinusoids) from the original signal.

Alternative/Complementary Method:

• MODWT-LSTM Prediction: For unknown task periods, use Maximal Overlap Discrete Wavelet Transform (MODWT) to decompose a long resting-state signal. Identify wavelets corresponding to low-frequency physiological noise (e.g., 5th to 9th level). Train a Long Short-Term Memory (LSTM) network on these wavelets to predict the noise during the subsequent task period, then subtract it [65].

Table 2: Performance of Selected Artifact Removal Methods

Method	Modality	Key Metric	Reported Performance	Primary Use Case
WPD-CCA [62]	fNIRS	ΔSNR (dB)	16.55 dB	Motion Artifact Removal
		Artifact Reduction (η)	41.40%
WPD-CCA [62]	EEG	ΔSNR (dB)	30.76 dB	Motion Artifact Removal
		Artifact Reduction (η)	59.51%
Motion-Net (CNN) [63]	EEG	ΔSNR (dB)	20.00 dB	Motion Artifact Removal
		Artifact Reduction (η)	86.00%
RLSE with SS [61]	fNIRS	Contrast-to-Noise Ratio	Significant improvement vs. Kalman filter & ICA	Physiological & Superficial Noise
PCA/GLM with SS [66]	fNIRS	Contrast-to-Noise Ratio	Superior to minimal preprocessing & 3 other methods	Physiological Noise in whole-head montage

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 3: Key Materials and Tools for Artifact Management in EEG-fNIRS Research

Item	Function/Description	Example Use Case
Short-Separation fNIRS Optodes	Source-detector pairs placed <1 cm apart to measure systemic hemodynamic noise from the scalp without cortical contribution [66] [64].	Used as a regressor to remove global physiological noise from standard long-separation channels.
Tri-axial Accelerometer	A sensor attached to the participant's head to provide objective, ground-truth measurement of head movement dynamics [59].	Used to identify motion artifact segments and validate motion correction algorithms.
Recursive Least-Squares Estimator (RLSE)	An adaptive filtering algorithm that efficiently estimates the parameters of a linear model, useful for separating signal from noise [61].	Modeling and removing physiological noise and superficial signals in fNIRS data.
Wavelet Packet Decomposition (WPD)	A signal processing method that provides a more detailed time-frequency representation than standard wavelet transform [62].	Decomposing signals to isolate motion artifact components for removal, especially in combination with CCA.
Mutual Information Feature Selection	A filter-based feature selection method that maximizes relevance and minimizes redundancy between features from different modalities [4].	Optimizing feature sets for EEG-fNIRS data fusion by selecting the most complementary and informative features for classification.

Effective artifact management is a critical, non-negotiable step in multimodal EEG-fNIRS research, directly impacting the validity of downstream data fusion and analysis. The protocols outlined herein provide a structured framework for tackling the primary challenges of motion, physiological, and environmental noise. By leveraging a combination of hardware solutions (e.g., short-separation channels, accelerometers) and advanced algorithmic approaches (e.g., WPD-CCA, RLSE, deep learning), researchers can significantly enhance data quality. This, in turn, ensures that subsequent fusion techniques uncover genuine neurovascular coupling dynamics, thereby increasing the reliability and interpretability of results in basic neuroscience and applied drug development studies.

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) provides a powerful multimodal approach for neuroscience research, combining EEG's millisecond-level temporal resolution with fNIRS's superior spatial localization of hemodynamic responses [67] [6]. However, this integration presents significant hardware challenges, particularly regarding accurate sensor placement, mechanical stability, and signal quality assurance. Traditional textile caps with manually prepared layouts often result in imprecise sensor positioning, leading to increased experimental variability [68]. This application note examines advanced hardware integration solutions, focusing on customizable 3D-printed headgear and integrated helmet systems that address these critical challenges for high-quality EEG-fNIRS data acquisition.

State of the Art in Integrated Headgear Systems

Recent advancements in neurotechnology have produced several innovative approaches to EEG-fNIRS integration. The table below summarizes the key performance characteristics of current hardware solutions.

Table 1: Performance Comparison of Integrated EEG-fNIRS Headgear Systems

System Name	Customization Approach	Key Technical Features	Reported Placement Accuracy	Supported Modalities
ninjaCap [68] [69]	3D-printed using atlas-based or subject-specific head models	Spring-relaxation algorithm for 3D-to-2D coordinate flattening; TPU flexible filament	2.2 ± 1.5 mm	fNIRS, EEG, DOT, oMEG
Chameleon-1 [70]	Modular, adaptable headband design	Electrodes at F3/F4 positions (10-20 system); adjustable sizing	Not explicitly quantified	EEG
HEFMI-ICH Hybrid Cap [31]	Pre-configured hybrid design	32 EEG electrodes, 32 fNIRS sources, 30 detectors; 90 fNIRS channels	Not explicitly quantified	EEG-fNIRS simultaneous recording

The ninjaCap system represents a significant advancement through its fully customizable, 3D-printed design that enables accurate translation of 3D brain coordinates to 2D printable panels [68]. This approach addresses fundamental limitations of traditional EEG caps when used for fNIRS studies, particularly the suboptimal clustering of channel distances around typical fNIRS target separations [68]. The system utilizes thermoplastic polyurethane (TPU) filaments like ninjaFlex, offering the necessary flexibility, durability, and comfort for research applications.

The ninjaCap Workflow: From Digital Model to Physical Cap

The generation of a customized ninjaCap follows a structured workflow that ensures precise sensor placement tailored to individual research requirements and anatomical considerations.

Table 2: ninjaCap Generation Workflow Components

Workflow Stage	Key Components	Output
Head Model & Probe Design	Colin 27 head model (default) or subject-specific MRI/photogrammetric models; Probe.SD file containing 3D node positions, anchor points, and grommet types	Registered probe layout on 3D head surface
3D-to-2D Coordinate Flattening	Spring-relaxation algorithm; hex lattice structures for uniform deformation; 10-5 EEG coordinate reference	2D panels representing flattened head surface sections
3D Modeling & Printing	Blender3D software for extrusion and assembly; TPU flexible filament material; panel splitting for print bed compatibility	Set of 3-4 .STL files representing cap panels
Physical Assembly	Ultrasonic welding for panel connection; sensor and optode integration	Complete functional headgear

The following diagram illustrates the complete ninjaCap generation workflow:

Research Reagent Solutions: Essential Materials for EEG-fNIRS Integration

Successful implementation of integrated EEG-fNIRS systems requires specific materials and components. The table below details essential research reagents and their functions.

Table 3: Essential Research Reagents for Custom EEG-fNIRS Headgear

Component Category	Specific Examples	Function & Application Notes
3D Printing Materials	ninjaFlex TPU [68]	Flexible filament providing necessary stretchability, durability, and comfort for headgear
EEG Electrodes	Ag-AgCl (Silver/Silver Chloride) disks [71]	Standard wet electrodes for high-quality signal acquisition; require conductive gel
fNIRS Optodes	Sources & detectors (e.g., NIRScout system [32])	Optical components for emitting and detecting near-infrared light through cerebral tissue
Hybrid Caps	Custom-designed EEG-fNIRS caps [31]	Integrated systems with pre-configured electrode and optode layouts for simultaneous recording
Conductivity Solutions	Electrolyte gels [70]	Ensure optimal electrical contact between EEG electrodes and scalp for signal quality
Adhesives & Stabilizers	Ultrasonic welding [68]	Panel connection method providing secure bonds without compromising material flexibility

Experimental Protocol: Implementing the ninjaCap System

Pre-Experimental Planning Phase

• Head Model Selection: Choose between atlas-based models (e.g., Colin 27) or subject-specific models derived from structural MRI or photogrammetric scans [68]. For population studies, use the Colin 27 model with linear scaling based on head circumference (HC) measurements.

• Probe Layout Design: Using AtlasViewer software, design the optode and electrode arrangement based on target brain regions. Specify source-detector distances according to fNIRS requirements (typically 20-40mm for adequate penetration depth). Save the configuration as a Probe.SD file containing 3D node positions, anchor points, and grommet type specifications [68].

• Cap Generation Parameters: Input the Probe.SD file and head circumference measurement into the cloud-based pipeline at bfnirs.openfnirs.org. The system will output a set of 3-4 .STL files representing the cap panels optimized for standard 3D printer beds [68].

Manufacturing and Assembly Phase

• 3D Printing Configuration: Use a fused deposition modeling (FDM) 3D printer with flexible TPU filament. Configure settings for optimal flexibility: 100% infill density, 0.15-0.20mm layer height, and slow print speed (20-30mm/s) to ensure quality [68].

• Post-Processing: Remove support material carefully to avoid damaging the flexible structure. Inspect each panel for printing defects that might affect sensor placement accuracy.

• Panel Assembly: Connect the printed panels using ultrasonic welding, ensuring precise alignment at joining points. Verify the complete assembly matches the intended head circumference before sensor integration [68].

Experimental Implementation Phase

• Sensor Integration: Install fNIRS optodes and EEG electrodes into their designated grommets on the assembled cap. For EEG, apply conductive gel according to standard protocols to achieve impedance below 10 kΩ [32].

• Subject Fitting: Position the cap on the participant's head using anatomical landmarks (nasion, inion, preauricular points) as reference. Ensure snug but comfortable fit to minimize motion artifacts.

• Signal Quality Verification: Perform pre-experimental signal checks for both modalities. For fNIRS, verify signal-to-noise ratio and detector sensitivity. For EEG, confirm impedance values and check for abnormal noise patterns [31].

Integration with Data Fusion Methodologies

Proper hardware integration directly enables advanced data fusion techniques for EEG-fNIRS signals. The complementary nature of these modalities—with EEG capturing rapid neural electrical activity and fNIRS tracking slower hemodynamic responses—creates powerful opportunities for multimodal brain state decoding [6]. The precise colocation of sensors ensured by systems like the ninjaCap facilitates accurate cross-modal correlation analysis and neurovascular coupling investigation [68] [67].

Advanced fusion approaches include model-based architectures like the Multi-Branch Convolutional Neural Network with Attention (MBC-ATT), which processes EEG and fNIRS signals through separate branches before applying cross-modal attention mechanisms for feature discrimination [6]. Such methods typically achieve 5-10% improvement in classification accuracy compared to unimodal systems [31], but depend heavily on precise spatial alignment of acquisition hardware.

The hardware solutions described herein provide the foundation for exploiting these analytical advantages by ensuring consistent, accurate sensor placement across experimental sessions and between subjects—a critical requirement for reproducible multimodal research.

In concurrent electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) studies, precise temporal synchronization of the two data streams is not merely beneficial—it is a fundamental prerequisite for valid data fusion and interpretation. These modalities capture distinct yet complementary physiological processes: EEG measures postsynaptic electrical potentials with millisecond temporal resolution, while fNIRS tracks hemodynamic responses related to blood oxygenation changes over seconds [72] [1]. This inherent difference in temporal dynamics means that without precise synchronization, any integrated analysis lacks temporal correspondence, potentially leading to misinterpretations of neurovascular coupling and brain function.

The core challenge lies in aligning these fundamentally different signals—one representing direct neuronal firing and the other reflecting metabolic consequences—to establish a meaningful relationship between electrical brain activity and hemodynamic responses. Furthermore, both data streams must be accurately synchronized with experimental events, ensuring that stimulus onsets and behavioral responses can be precisely mapped to both electrical and hemodynamic brain responses [73]. The selection of an appropriate synchronization strategy consequently directly impacts the validity and reliability of research findings in multimodal brain imaging.

Synchronization Methodologies

Hardware-Based Triggering

Hardware synchronization establishes a direct physical connection between devices, creating a unified timekeeping system that is both robust and precise. This approach typically utilizes specialized devices that generate or relay trigger signals recognized simultaneously by both EEG and fNIRS acquisition systems.

• Dedicated Synchronization Devices: Instruments like the PortaSync (a wireless, handheld device) offer a versatile hardware solution. They typically connect to the fNIRS software via Bluetooth and feature analog inputs and outputs. These can receive external trigger signals or generate their own, which are then recorded in the neuroimaging software while simultaneously sending the same signal to the EEG amplifier via its analog input [73]. Another example is the LabStreamer, which provides multiple high-speed analog inputs (up to 10kHz) suitable for capturing very fast-paced signals and connects to acquisition software via the Lab Streaming Layer protocol [73].

• Parallel Port Triggers: A traditional yet effective method involves using the parallel port of a computer controlling the stimulus presentation software. The output signal can be split and sent to both the EEG amplifier and, through a device like the PortaSync, to the fNIRS system. This ensures both systems receive identical event markers [73].

• Analog Output to Input Connection: In simpler setups, a direct cable can connect the analog output of one system (e.g., the stimulus computer or EEG amplifier) to the analog input of the other (the fNIRS system, or vice versa). The voltage change in the signal stream precisely marks the event onset in both recordings.

Table 1: Key Hardware Synchronization Equipment

Equipment	Primary Function	Key Characteristics	Compatibility Notes
PortaSync	Generates/receives sync triggers	Wireless, Bluetooth, analog I/O, buttons for manual marking	Connects to OxySoft; analog out to EEG [73]
LabStreamer	High-speed data acquisition	Four analog inputs (up to 10kHz), LSL connection	Suitable for very fast EEG signals [73]
Parallel Port Sync Cable	Transmits TTL pulses	Direct computer connection, requires signal splitting	Classic, reliable method for sending triggers [73]

Software-Based Alignment

Software synchronization relies on network communication protocols to achieve temporal alignment, eliminating the need for physical cables between systems while maintaining high precision when properly configured.

• Lab Streaming Layer (LSL): This open-source ecosystem represents the current gold standard for software-based synchronization. LSL enables different software applications and acquisition devices on the same network to stream time-stamped data, including triggers and actual brain signals. The LSL system automatically handles the time-stamping, allowing the fNIRS software (e.g., OxySoft) to receive external event and data streams from EEG software or stimulus presentation packages [73] [74]. This framework is particularly advantageous for integrating multiple data streams beyond just triggers, such as simultaneous EEG and fNIRS signal acquisition into a single software.

• Distributed Component Object Model (DCOM): This software interface allows different applications on the same computer to communicate. Stimulus presentation software can use DCOM to send triggers directly to the fNIRS software (e.g., OxySoft) without additional hardware [73]. This method is computer-dependent but can be highly effective for specific software combinations.

The following diagram illustrates the decision-making workflow for selecting the appropriate synchronization method based on equipment capabilities:

Comparative Analysis of Synchronization Approaches

Each synchronization method offers distinct advantages and limitations, making them suitable for different experimental scenarios and equipment constraints. The choice between hardware and software approaches depends on multiple factors, including required precision, equipment compatibility, setup complexity, and experimental environment.

Table 2: Hardware vs. Software Synchronization Comparison

Feature	Hardware Triggering	Software-Based Alignment
Primary Mechanism	Physical cables (analog, parallel port)	Network protocols (LSL, DCOM)
Typical Precision	Very high (sub-millisecond)	High (millisecond)
Setup Complexity	Moderate (requires cabling)	Low (requires network/software config.)
Key Advantage	Robustness, independence from OS delays	Flexibility, no additional cables
Key Limitation	Additional equipment cost, physical setup	Potential for software/network latency
Stimulus Locking	Excellent	Excellent
Best For	Environments with electrical noise; simple, robust setups	Flexible labs; streaming multimodal data beyond triggers

Detailed Experimental Protocols

Protocol A: Hardware Synchronization Using PortaSync

This protocol provides a step-by-step methodology for achieving robust synchronization using the PortaSync device, suitable for experiments where software streaming is not feasible.

1. Equipment Setup and Connections:

• Connect the PortaSync to the computer running the fNIRS acquisition software (e.g., OxySoft) via Bluetooth.
• Use the analog output of the PortaSync and connect it to an analog input on the EEG amplifier using an appropriate cable.
• Ensure the stimulus presentation computer is configured to send triggers. If using a parallel port, connect it to the analog input of the PortaSync using a parallel sync cable [73].

2. Signal Configuration:

• In the fNIRS software, verify the Bluetooth connection and confirm that the PortaSync is recognized as an input device.
• In the EEG acquisition software, configure the specific analog input channel to record the trigger signal from the PortaSync. Set an appropriate voltage threshold to detect trigger onsets accurately.

3. Experimental Recording and Validation:

• Begin recording both fNIRS and EEG systems.
• Initiate the stimulus paradigm. Each stimulus event should send a trigger via the PortaSync, which will be simultaneously recorded in both the fNIRS data (via Bluetooth) and the EEG data (via the analog cable).
• For validation, include a series of unique test triggers at the beginning of the recording (e.g., using the PortaSync's buttons). Visually inspect the aligned data offline to confirm the trigger timestamps in both data streams match precisely.

Protocol B: Software Synchronization Using Lab Streaming Layer (LSL)

This protocol outlines the procedure for implementing LSL-based synchronization, ideal for modern setups that benefit from cable-free operation and integrated data handling.

1. Network and Software Preparation:

• Install the LSL library on all computers involved in the experiment (stimulus, EEG, fNIRS).
• Ensure all computers are connected to the same local network with minimal latency.
• Verify that the EEG acquisition software and the stimulus presentation software (e.g., PsychoPy, Presentation) support LSL output streams.

2. Stream Configuration:

• Configure the stimulus software to stream event markers (e.g., "StimulusOnset", "BlockStart") as an LSL outlet.
• Configure the fNIRS software (e.g., OxySoft) to receive these external LSL streams [73].
• Optional: Configure the EEG software to stream the raw EEG data as an LSL outlet, which can also be received and recorded by the fNIRS software for a more integrated data structure [73].

3. Recording and Synchronization Check:

• Start the LSL stream from the stimulus computer and then from the EEG system.
• Start the recording in the fNIRS software, ensuring it is subscribing to the correct LSL streams for triggers and/or EEG data.
• Run the experiment. The fNIRS software will record its own data while simultaneously incorporating the time-stamped triggers and other data from the network streams.
• Offline, the shared LSL time stamps provide the basis for aligning the fNIRS and EEG data files with high precision.

The Scientist's Toolkit: Essential Research Reagents and Materials

Successful execution of synchronized EEG-fNIRS experiments requires specific hardware and software components. The following table details the essential "research reagents" and their critical functions in the experimental workflow.

Table 3: Essential Materials for Synchronized EEG-fNIRS Research

Item	Category	Critical Function	Example Models/Protocols
Synchronization Hardware	Equipment	Generates or transmits precise physical trigger signals	PortaSync, LabStreamer [73]
Lab Streaming Layer (LSL)	Software	Enables software-based, time-stamped data streaming on a network	Open-source LSL library [73] [74]
Integrated EEG-fNIRS Cap	Consumable/Material	Holds both sensor types in a stable, co-registered spatial layout	actiCAP Xpress Twente Medical Systems, custom 3D-printed helmets [5] [74]
Stimulus Presentation Software	Software	Presforms experiments and generates event markers for synchronization	PsychoPy, Presentation, E-Prime
Analog Cables & Connectors	Accessory	Transmits analog trigger signals between devices	BNC, DIN, or other system-specific cables
Data Fusion Analysis Toolbox	Software	Performs joint analysis of synchronized multimodal data	MATLAB toolboxes, Python (MNE), Homer3, NIRS-KIT

The strategic selection and proper implementation of synchronization techniques form the bedrock of any high-quality concurrent EEG-fNIRS study. Hardware triggering offers a physically robust solution, often preferred for its consistent precision and simplicity. In contrast, software-based alignment via frameworks like LSL provides immense flexibility and is well-suited to complex, multi-stream experimental designs. The optimal choice is not universal but must be determined by the specific research question, the available equipment's capabilities, and the constraints of the experimental environment. By adhering to the detailed protocols and utilizing the essential tools outlined in this document, researchers can ensure the temporal integrity of their multimodal data, thereby creating a reliable foundation for advanced data fusion and the generation of meaningful insights into brain function.

The accurate integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) data represents a powerful approach in neuroscience for investigating brain function across electrophysiological and hemodynamic domains. The cornerstone of effective multimodal data fusion is the precise co-registration of sensors to underlying brain anatomy. The International 10-20 system provides a standardized framework for electrode placement, but its effective implementation requires careful consideration of anatomical correlations, especially when integrating fNIRS optodes. This application note details standardized protocols for co-registering EEG and fNIRS sensors within the 10-20 framework, ensuring optimal data quality for subsequent fusion techniques in both clinical and research settings. The precise alignment of sensor positioning with cortical landmarks is fundamental to maximizing the spatial accuracy and interpretative power of multimodal brain imaging studies [75] [5].

Anatomical and Technical Foundations

The International 10-20 system standardizes sensor placement based on proportional distances between key cranial landmarks (nasion, inion, and preauricular points), ensuring consistent positioning across subjects despite variations in head size and shape. Understanding the relationship between these external scalp positions and underlying cerebral structures is critical for accurate source localization and interpretation of recorded signals.

Anatomical Correlations Across Development

Neuroimaging studies have validated the general consistency between 10-20 positions and their underlying cerebral structures. However, important developmental differences exist, particularly in infant populations. Research using MRI has demonstrated that while 10-20 electrode placements in infants generally project to the same cerebral structures as in adults, notable differences exist for occipital (O1, O2) and temporal (T5, T6) channels, which project over lower anatomical structures in infants [75]. This underscores the importance of age-specific anatomical atlases for developmental studies. Furthermore, anatomical asymmetries, such as the Yakovlean torque which displaces the right sylvian fissure dorsally and frontally, can result in sensors P3 and P4 being at different distances from this structure, highlighting the need for individualized co-registration where possible [75].

Table 1: Key Differences in 10-20 System Correlations Between Infants and Adults

Parameter	Infant Population	Adult Population
O1/O2 Projection	Over lower anatomical structures	Standard projection patterns
T5/T6 Projection	Over lower anatomical structures	Standard projection patterns
Template Availability	Infant-specific probabilistic templates	Standard MNI adult templates
Sulcal Variability	Higher variability in folding patterns	More consistent sulcal patterns

Technical Complementarity of EEG and fNIRS

EEG and fNIRS offer complementary information about brain activity, making them ideal partners for multimodal fusion. EEG provides millisecond-level temporal resolution of electrical neural activity but suffers from relatively low spatial resolution and sensitivity to artifacts. Conversely, fNIRS measures hemodynamic responses with better spatial resolution and robustness to motion artifacts, but has significantly slower temporal resolution due to the inherent latency of neurovascular coupling [5] [6]. This technical complementarity enables more comprehensive brain monitoring when the signals are properly integrated, but necessitates precise sensor co-registration to ensure spatial correspondence between the different modalities.

Sensor Integration and Co-registration Protocols

Integrated Helmet Design Considerations

Effective co-registration begins with physical sensor integration. Current approaches typically combine EEG electrodes and fNIRS optodes within a single acquisition helmet. Several design strategies exist:

• Flexible Cap Integration: EEG electrode caps serve as a substrate with punctures for fNIRS probe fixtures [5]. While straightforward to implement, elastic fabrics may yield inconsistent optode-scalp contact pressure and variable source-detector distances across subjects.
• Customized Rigid Helmets: 3D-printed or thermoplastic helmets offer subject-specific customization, maintaining consistent optode positioning and improving scalp coupling [5]. These approaches reduce movement artifacts but incur higher costs and manufacturing time.

The integration design must minimize crosstalk between modalities—ensuring that EEG electrodes do not interfere with optical paths and that fNIRS hardware does not introduce electrical noise into EEG recordings [76].

Anatomical Co-registration Workflow

Precise alignment of sensor positions with individual anatomy significantly enhances the spatial accuracy of multimodal measurements. The following protocol outlines a comprehensive co-registration procedure:

Diagram 1: Anatomical Co-registration Workflow for Multimodal Setup

• Subject Preparation and Landmark Identification: Precisely measure and mark standard 10-20 landmarks (nasion, inion, left/right preauricular points) and the vertex (Cz) on the subject's scalp. Ensure hair is parted to expose measurement points.
• Sensor Placement: Apply the integrated EEG-fNIRS cap according to manufacturer guidelines, aligning reference sensors with anatomical landmarks. For custom setups, position sensors according to the expanded 10-10 or 10-5 systems for higher spatial coverage [77].
• Position Digitization: Use a 3D digitization system (e.g., electromagnetic or optical digitizer) to record the precise spatial coordinates of each EEG electrode, fNIRS optode (sources and detectors), and cranial landmarks. This step is crucial for moving from standardized positions to actual sensor locations.
• Anatomical Reference Acquisition: Obtain structural MRI data aligned with the sensor positions. When individual MRI is unavailable, use age-appropriate template brains (e.g., MNI template for adults, infant templates for developmental studies) [75] [78].
• Coordinate Transformation: Coregister the digitized sensor positions with the anatomical dataset (individual or template MRI). This involves computing a transformation matrix (including translation, rotation, and scaling parameters) to align the two coordinate systems [78].
• Head Model Generation: Construct a forward model describing how electrical currents (for EEG) and light propagation (for fNIRS) travel through head tissues. Boundary Element Method (BEM) or Finite Element Method (FEM) models constructed from segmented MRI data (skin, skull, CSF, cortex) provide more accurate results than spherical models [78].

Table 2: Coregistration Methods and Their Applications

Method	Procedure	Accuracy	Best Suited For
Landmark-based	Aligns sensors using nasion, inion, preauricular points, and Cz.	Moderate	Standardized studies without individual MRI
3D Digitization	Records exact 3D coordinates of each sensor and landmark.	High	Studies requiring individual head models
Template Fitting	Warps standard sensor positions to fit individual head shape.	Moderate-High	Large cohort studies with structural MRI
Photogrammetry	Uses multiple 2D images to reconstruct 3D sensor positions.	High	Rapid acquisition without specialized digitizers

Optimization and Validation Protocols

Sensor Configuration Optimization

Beyond standardized placement, optimizing electrode subsets for specific neural sources can significantly enhance signal quality while reducing system complexity. Genetic algorithms offer a powerful approach for this optimization:

Genetic Algorithm Optimization Protocol:

• Define Optimization Objectives: Formulate a multi-objective optimization problem that simultaneously (1) minimizes source localization error and (2) minimizes the number of electrodes required [77].
• Initialize Population: Generate an initial population of random electrode subsets (chromosomes).
• Fitness Calculation: For each subset, compute the leadfield matrix and solve the inverse problem using methods like weighted minimum norm estimation (wMNE) or sLORETA. Calculate localization error against known ground-truth sources [77].
• Evolutionary Operations: Apply selection, crossover, and mutation operations across generations to evolve increasingly optimal sensor configurations.
• Solution Extraction: Identify non-dominated solutions (Pareto front) representing optimal trade-offs between channel count and localization accuracy.

Research demonstrates that optimized low-density electrode subsets (e.g., 6-8 electrodes) can achieve accuracy comparable to full high-density arrays (200+ channels) for single-source localization tasks, highlighting the potential for more efficient experimental designs [77].

Experimental Validation Framework

Rigorous validation ensures that co-registration accuracy translates to improved experimental outcomes across different application domains:

Table 3: Validation Protocols for Specific Application Domains

Application Domain	Validation Task	Expected Outcome	Quality Metrics
Motor Imagery	Repetitive hand open/close, wrist flexion/extension motor imagery or execution [32].	Contralateral primary motor cortex activation (EEG: µ/beta ERD; fNIRS: HbO increase).	Spatial consistency between EEG source localization and fNIRS activation focus.
Working Memory	n-back tasks with varying cognitive load (0-back, 2-back, 3-back) [6].	Prefrontal cortex activation scaling with task difficulty.	Correlation between EEG frontal theta power and fNIRS HbO in DLPFC.
Language Processing	Word generation or semantic decision tasks [3].	Left-lateralized frontal-temporal activation.	Laterality index consistency between modalities.
Epilepsy	Interictal spike or seizure recording [5].	Concordant localization of epileptiform activity.	Spatial overlap between EEG spike source and fNIRS hemodynamic response.

For all validation experiments, simultaneously record EEG and fNIRS data during task performance. Preprocess signals according to modality-specific pipelines (filtering, artifact removal), then analyze spatial concordance between EEG source reconstructions and fNIRS activation maps. Quantitative metrics should include distance between activation centroids, spatial correlation of activation maps, and temporal correlation between EEG features (e.g., band power) and fNIRS hemodynamic responses [5] [6].

The Scientist's Toolkit: Research Reagents and Materials

Table 4: Essential Materials for EEG-fNIRS Co-registration Research

Item	Specification/Example	Primary Function
EEG Acquisition System	64+ channel systems (e.g., Neuroscan SynAmps2) [32].	Records electrical brain activity with high temporal resolution.
fNIRS Acquisition System	Multichannel systems (e.g., NIRScout with 8+ sources/detectors) [32].	Measures hemodynamic responses via light absorption.
Integrated Caps	Flexible caps with pre-configured EEG/fNIRS layouts or custom 3D-printed helmets [5].	Ensures stable, consistent sensor placement.
3D Digitizer	Electromagnetic (e.g., Polhemus) or optical (e.g., Structure Sensor) systems.	Records precise 3D coordinates of sensors and landmarks.
Head Model Software	EEGLAB, FieldTrip, SPM, NIRS Brain AnalyzINBOX [78].	Creates forward models for source reconstruction.
Anatomical Templates	Age-appropriate templates (e.g., MNI for adults, infant templates) [75].	Provides anatomical reference when individual MRI is unavailable.
Data Fusion Platforms	Custom MATLAB/Python scripts with toolboxes for multimodal analysis.	Implements feature-level or decision-level fusion algorithms.

Diagram 2: System Components for Multimodal Co-registration

Precise co-registration of EEG and fNIRS sensors within the International 10-20 system framework is a critical prerequisite for meaningful multimodal data fusion. By implementing the anatomical coregistration protocols, optimization strategies, and validation frameworks outlined in this document, researchers can significantly enhance the spatial accuracy and interpretative power of their neuroimaging studies. Future directions in this field include the development of more sophisticated integrated helmet designs that ensure consistent scalp coupling, advances in real-time coregistration using computer vision, and the creation of age-specific and disease-specific head models to further improve source localization accuracy across diverse populations.

Addressing Data Imbalance and Improving Model Generalization

In multimodal electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) research, data imbalance and poor model generalization present significant barriers to developing robust brain-computer interfaces (BCIs) and clinical diagnostic tools. The inherent differences in signal characteristics between electrophysiological (EEG) and hemodynamic (fNIRS) measurements create natural asymmetries that conventional machine learning approaches struggle to reconcile. EEG captures millisecond-level electrical neural activity with high temporal resolution but limited spatial precision, while fNIRS measures hemodynamic changes through oxygenation with superior spatial resolution but slower temporal dynamics [7] [79]. This fundamental discrepancy often leads to imbalanced feature learning during model training, where one modality dominates the optimization process, thereby limiting the synergistic potential of multimodal integration [79].

The problem extends beyond technical implementation to practical research constraints. Multimodal EEG-fNIRS datasets remain scarce, with limited public availability creating additional generalization challenges [7]. Furthermore, the temporal misalignment between rapid electrical signals and slower hemodynamic responses introduces representation learning complexities that unsupervised symmetric techniques attempt to resolve [7]. These challenges necessitate specialized frameworks that explicitly address imbalance while enhancing cross-subject and cross-task generalization, particularly in real-world applications such as Parkinson's disease detection [80], motor imagery classification [32], and semantic decoding [81].

Technical Framework for Balanced Multimodal Learning

Dual-Decoder Architecture for Feature Disentanglement

A promising approach to addressing data imbalance involves feature decoupling through a dual-decoder architecture that separates modality-general and modality-specific features [79]. This framework employs dedicated global feature encoders for EEG and fNIRS domains, followed by two distinct decoders:

• Query Decoders: Focus on learning modality-general features by aligning EEG and fNIRS data in a shared subspace using Center Moment Distance (CMD) based distributional alignment [79]
• Key Decoders: Isolate modality-specific features that capture unique neural and vascular characteristics, preserving distinctive information often disregarded as noise [79]

This architectural approach mitigates signal interference while enhancing extraction of task-relevant information. The separation allows models to leverage complementary strengths of both modalities - the temporal precision of EEG alongside the spatial specificity of fNIRS - without allowing either modality to dominate the learning process [79].

Gradient Rebalancing Strategy

To explicitly counter learning imbalance, a Gradient Rebalancing (GradReb) strategy actively monitors and regulates discriminative differences between modalities during training [79]. This approach:

• Attenuates gradient contributions from the dominant modality during backpropagation
• Ensures balanced optimization across both EEG and fNIRS feature extractors
• Prevents one modality from becoming over-represented in the final model [79]

Implementation results demonstrate that GradReb significantly improves joint learning framework performance, particularly for classification tasks where baseline methods typically favor one modality [79].

Representation Learning for Few-Shot Scenarios

For limited data scenarios, multimodal representation learning provides an alternative pathway to improved generalization. The EEG-fNIRS Representation Model (EFRM) employs a two-stage approach: [82]

• Pre-training: Learns both modality-specific and shared representations across EEG and fNIRS using large-scale unlabeled data
• Transfer Learning: Adapts pre-trained models to specific downstream tasks with minimal labeled examples [82]

This method enables effective adaptation to single-modality datasets when both modalities aren't available, enhancing practical flexibility. By leveraging the shared domain across EEG and fNIRS, this approach outperforms single-modality methods, especially for fNIRS classification where traditional transfer learning has shown limitations [82].

Table 1: Comparative Analysis of Technical Approaches for Addressing Data Imbalance

Approach	Core Mechanism	Advantages	Implementation Considerations
Dual-Decoder Architecture [79]	Feature decoupling into modality-general and specific components	Reduces signal interference; Enhances task-relevant information extraction	Requires careful loss function balancing; Increases model complexity
Gradient Rebalancing (GradReb) [79]	Regulation of gradient flow during training	Prevents modality dominance; Ensures balanced feature learning	Needs monitoring of discriminative differences between modalities
Multimodal Representation Learning [82]	Two-stage pre-training and transfer learning	Effective for few-shot scenarios; Works with limited labeled data	Requires substantial unlabeled data for pre-training phase

Experimental Protocols for Imbalance-Aware Research

Simultaneous EEG-fNIRS Data Acquisition Protocol

Proper data collection forms the foundation for addressing imbalance and generalization challenges. The following protocol ensures high-quality, temporally aligned multimodal data: [83] [84]

• Equipment Setup: Use synchronized EEG and fNIRS systems with trigger synchronization. For EEG, employ 64-channel caps with left mastoid (M1) reference placement, impedance maintained below 10 kΩ. For fNIRS, configure sources and detectors according to international 10-5 systems, with source-detector distances of 3cm for cortical sensitivity [32] [81]
• Preprocessing Pipeline:
  • EEG: Apply 0.5-100Hz bandpass filtering, 50Hz notch filtering for line noise removal, independent component analysis (ICA) for ocular and motion artifacts [32]
  • fNIRS: Convert raw light intensity to optical density, then to hemoglobin concentrations (HbO, HbR) using Modified Beer-Lambert Law. Perform motion correction using wavelet or spline interpolation methods [80] [32]
• Temporal Alignment: Implement hardware-synchronized triggers between stimulation paradigms and data acquisition systems to ensure precise timing across modalities [83]

Cross-Validation Strategies for Generalization Assessment

Robust evaluation methodologies are essential for properly assessing model generalization: [80]

• Stratified Leave-One-Subject-Out (LOSO) Cross-Validation: Ensures subject-independent evaluation while maintaining class balance in training folds
• Nested Cross-Validation: Employs an inner loop for hyperparameter tuning and an outer loop for performance estimation to prevent optimistic bias
• Modality Ablation Studies: Systematically evaluate performance using EEG-only, fNIRS-only, and combined modalities to quantify integration benefits [79]

Data Augmentation for Addressing Dataset Limitations

To combat limited dataset sizes and class imbalances, employ multimodal data augmentation: [32]

• Temporal Transformations: Apply small temporal shifts (±100ms) to account for hemodynamic response latency variations
• Signal Perturbation: Introduce controlled white noise at signal-to-noise ratios above 20dB to improve robustness [32]
• Synthetic Data Generation: For motor imagery paradigms, generate realistic synthetic HD-fNIRS-EEG datasets that simulate tasks with known ground truth for method validation [7]

Application Case Studies

Parkinson's Disease Detection

In Parkinson's disease detection, fNIRS data from the frontopolar cortex (FPC) region can be leveraged with machine learning to achieve 85% accuracy using Support Vector Machines (SVM) [80]. The experimental protocol involved:

• Participants: 120 Parkinson's patients (Hoehn and Yahr stages 1-2) and 60 healthy controls
• Data Acquisition: ETG-4000 fNIRS system monitoring prefrontal cortex with 22 channels
• Feature Extraction: General linear model applied to cerebral blood oxygen changes, with β-values extracted for classification [80]

To address potential data imbalance between patient and control groups, the study employed SHapley Additive exPlanations (SHAP) to identify the most contributory channels (CH01, CH04, CH05, CH08), enabling feature selection that improved model generalization [80].

Motor Imagery Classification

For motor imagery tasks involving multiple joint movements, a balanced multimodal approach achieves 65.49% classification accuracy between hand open/close and shoulder pronation/supination using EEG data with deep learning (ShallowConvNet) and data augmentation [32]. The protocol included:

• Tasks: 8 motor imagery tasks from 4 joint types (hand, wrist, elbow, shoulder)
• Trial Structure: 2s cue presentation, 4s imagery period, 10-12s randomized rest period to accommodate hemodynamic response delays [32]
• Balanced Design: 40 trials per task across 18 subjects, totaling 5760 trials to ensure robust representation [32]

Table 2: Performance Comparison Across Application Domains

Application Domain	Modality	Key Methodology	Reported Performance	Generalization Approach
Parkinson's Detection [80]	fNIRS	SVM with channel selection	85% accuracy, F1=0.85, AUC=0.95	SHAP analysis for feature importance
Motor Imagery [32]	EEG-fNIRS	ShallowConvNet with augmentation	65.49% accuracy (hand vs. shoulder tasks)	Large-scale dataset (5760 trials)
Semantic Decoding [81]	EEG-fNIRS	Multimodal fusion	Differentiating animals vs. tools	Sensory-based imagery tasks
Brain State Characterization [83]	EEG-fNIRS	Joint learning framework	Improved spatial-temporal resolution	Standardized acquisition protocol

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 3: Research Reagent Solutions for EEG-fNIRS Studies

Item	Specification	Function/Application
EEG Acquisition System [32]	64-channel Neuroscan SynAmps2 amplifier, 1000Hz sampling	Records electrical brain activity with high temporal resolution
fNIRS Acquisition System [32]	NIRScout system (NIRx), 8 sources, 8 detectors, 7.8125Hz sampling	Measures hemodynamic responses through light absorption
Multimodal Caps [32]	Integrated EEG electrodes + fNIRS optodes arranged in 10-5 system	Enables simultaneous co-located measurement
ETG-4000 System [80]	Near-infrared brain function imaging instrument, 695nm & 830nm wavelengths	Specialized for hemodynamic monitoring in clinical applications
HD-DOT System [7]	High-density diffuse optical tomography with 16 sources, 12 detectors	Provides high spatial resolution for cortical mapping
Data Processing Tools [83]	MATLAB/Python with MNE-Python, NIRSKit, HomER2	Preprocessing, artifact removal, feature extraction
Synchronization Interface [85]	TTL trigger systems with LabView application	Ensures temporal alignment between modalities
Stimulus Presentation Software [86]	PsychToolbox, Presentation, OpenSesame	Controls experimental paradigms with precise timing

Visualizing the Experimental Workflow

Multimodal Data Acquisition and Processing Pipeline

Dual-Decoder Architecture for Balanced Feature Learning

Movement Artifact Mitigation in Naturalistic Study Environments

The paradigm in neuroimaging is shifting from controlled laboratory settings toward wearable, ubiquitous monitoring in naturalistic environments [87] [88]. This transition to ecologically valid research is crucial for understanding brain function during real-world activities but introduces significant challenges from motion artifacts (MA). These artifacts, caused by subject movement during data acquisition with wearable devices, can severely corrupt electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals, potentially rendering them unusable [87] [62]. For research relying on data fusion techniques from these complementary modalities—EEG providing millisecond temporal resolution and fNIRS offering better spatial localization and motion robustness [6] [29]—effective motion artifact mitigation becomes paramount. Without robust correction, subsequent fusion and interpretation risk significant error, undermining the validity of findings in cognitive neuroscience, clinical neurology, and drug development research [87] [29].

Motion Artifact Characteristics and Impact on Data Fusion

Motion artifacts manifest differently in EEG and fNIRS signals, presenting unique challenges for data fusion pipelines. In EEG, artifacts originate from muscle activity (EMG), electrode cable swings, and changes in electrode-skin impedance, leading to high-amplitude, high-frequency spikes and slow drifts that obscure neural signals [89] [29]. fNIRS artifacts arise from optode displacement, pressure changes, and altered scalp hemodynamics, causing baseline shifts, spikes, and increased variance in hemoglobin concentration measurements [90] [29].

The complementary nature of EEG and fNIRS artifacts complicates multimodal fusion. Artifacts are often temporally correlated with behavioral events but exhibit distinct morphological characteristics across modalities [29]. This differential contamination can desynchronize fused features, leading to misalignment in joint analysis and erroneous interpretation of neurovascular coupling relationships. Consequently, developing robust, integrated artifact mitigation protocols is essential for advancing data fusion techniques in naturalistic research [7] [29].

Quantitative Performance of Artifact Correction Methods

Traditional and Wavelet-Based Signal Processing Approaches

Traditional artifact correction methods include filtering, regression, and component analysis techniques. Recently, Wavelet Packet Decomposition (WPD) and its combination with Canonical Correlation Analysis (CCA) have demonstrated superior performance for single-channel denoising, a critical requirement for wearable systems with limited channels [87] [62].

Table 1: Performance of WPD and WPD-CCA for Motion Artifact Correction

Method	Modality	Best Performing Wavelet	ΔSNR (dB)	Artifact Reduction (η%)
WPD (Single-Stage)	EEG	db2	29.44	-
WPD (Single-Stage)	EEG	db1	-	53.48
WPD-CCA (Two-Stage)	EEG	db1	30.76	59.51
WPD (Single-Stage)	fNIRS	fk4	16.11	26.40
WPD-CCA (Two-Stage)	fNIRS	db1	16.55	-
WPD-CCA (Two-Stage)	fNIRS	fk8	-	41.40

The two-stage WPD-CCA method consistently outperforms single-stage WPD, with artifact reduction increasing by 11.28% for EEG and 56.82% for fNIRS when using the two-stage approach [62]. This demonstrates its particular effectiveness for fNIRS signals, making it highly suitable for multimodal studies where fNIRS signals are less inherently noisy but require cleaner baselines for accurate hemodynamic response interpretation [87].

Learning-Based Approaches

Machine and deep learning methods represent the cutting edge of artifact mitigation research, showing promise for adaptive, data-driven correction.

Table 2: Learning-Based Approaches for fNIRS Motion Artifact Correction

Method	Architecture	Key Mechanism	Reported Performance
Wavelet Regression ANN [90]	Artificial Neural Network	Unbalance index from entropy cross-correlation	Improved Contrast-to-Noise Ratio in gait tasks
MA Classifier [90]	LDA, SVM, KNN, GBT	Vigilance level detection during walking	Identified MA-induced accuracy reduction
U-Net HRF Reconstruction [90]	Convolutional Neural Network	Reconstruction of hemodynamic response	Lowest Mean Squared Error vs. wavelet/AR models
Denoising Auto-Encoder [90]	Auto-Encoder	Specialized loss function on synthetic data	Effective on open-access experimental datasets
sResFCNN + FIR Filter [90]	Fully Connected NN	Simplified residual architecture	Enhanced denoising capability

These learning-based approaches are particularly valuable for data fusion applications as they can be designed to process both EEG and fNIRS signals within a unified architecture, potentially learning cross-modal artifact patterns [90] [58]. For instance, DeepSyncNet represents a novel deep learning framework that integrates EEG and fNIRS at the feature extraction stage using Attentional Fusion, enhancing classification for motor imagery and mental arithmetic tasks in BCIs [58].

Experimental Protocols for Artifact Mitigation

Protocol 1: WPD-CCA for Single-Channel Denoising

Application: Correction of motion artifacts in single-channel EEG or fNIRS signals collected during naturalistic tasks.

Workflow:

• Signal Acquisition: Record single-channel EEG/fNIRS during experimental paradigm.
• Wavelet Packet Decomposition: Decompose signal using selected wavelet packet (db1 recommended for EEG, fk8 for fNIRS) to level 5.
• Node Selection: Identify and threshold nodes containing artifacts based on statistical parameters (e.g., kurtosis, entropy).
• Canonical Correlation Analysis: Apply CCA to denoised wavelet coefficients and original signal to extract artifact components.
• Signal Reconstruction: Reconstruct clean signal using inverse WPD on artifact-free coefficients.
• Validation: Calculate ΔSNR and η to verify performance against benchmark datasets.

Protocol 2: Naturalistic fNIRS Assessment with AIDE

Application: fNIRS data collection and analysis in real-world environments with unpredictable event timing.

Workflow:

• Experimental Setup: Configure portable fNIRS system for prefrontal cortex monitoring during natural tasks (e.g., laundry cycle, social media use) [88] [91].
• Data Collection: Record hemodynamic responses with concurrent behavioral logging.
• Motion Artifact Identification: Detect motion-contaminated segments using moving standard deviation or accelerometer data.
• AIDE Processing: Apply Automatic IDentification of functional Events to identify significant brain activation events without rigid epoching [88].
• CBSI Correction: Implement correlation-based signal improvement to reduce motion artifacts and physiological noise.
• Statistical Analysis: Compare activation patterns between experimental conditions using generalized linear models.

This protocol was successfully applied in a naturalistic study investigating olfactory stimuli during laundry tasks, where fNIRS with AIDE identified differential prefrontal activation between fragranced and unfragranced conditions despite the movement-rich environment [88].

Protocol 3: Deep Learning Artifact Removal

Application: Adaptive artifact correction for multimodal EEG-fNIRS data using deep learning architectures.

Workflow:

• Data Preparation: Curate labeled dataset of artifact-contaminated and clean signals (synthetic data generation possible via auto-regressive models).
• Network Selection: Choose appropriate architecture (U-Net, DAE, sResFCNN) based on signal characteristics.
• Model Training: Optimize network parameters using specialized loss functions combining reconstruction error and spectral coherence.
• Cross-Modal Attention: For multimodal data, implement attention mechanisms to weight feature importance across EEG and fNIRS streams [6] [58].
• Validation: Assess performance using ΔSNR, Mean Squared Error, and classification accuracy on held-out test sets.
• Deployment: Integrate trained model into real-time processing pipeline for online artifact correction.

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 3: Key Research Materials for Movement Artifact Mitigation Studies

Item	Function/Application	Example Specifications
Wearable EEG System	Mobile neural electrical activity recording	Science-grade, 8-64 channels, wireless capability [89]
Portable fNIRS Device	Hemodynamic response monitoring in natural settings	Continuous wave system, multiple wavelengths, wearable design [88] [29]
Motion Tracking System	Quantification of subject movement for artifact reference	IMU sensors, accelerometers, gyroscopes [62]
Wavelet Toolbox	Signal decomposition for WPD-based artifact removal	MATLAB, Python PyWavelets, with multiple wavelet families [87]
Deep Learning Framework	Implementation of neural networks for artifact correction	TensorFlow, PyTorch with custom architectures [90] [58]
Benchmark Datasets	Method validation and performance comparison	Publicly available EEG-fNIRS datasets with motion artifacts [62] [58]
AIDE Software Platform	Automatic identification of functional events in naturalistic fNIRS data	fNIRS analysis toolbox for naturalistic paradigms [88]
Homer2/3 Toolkit	Standard fNIRS processing including motion correction	MATLAB-based environment for fNIRS analysis [90]

Effective motion artifact mitigation is foundational for successful EEG-fNIRS data fusion in naturalistic research environments. The quantitative evidence demonstrates that two-stage approaches like WPD-CCA significantly outperform single-stage methods, particularly for fNIRS signals. Emerging deep learning architectures show promise for adaptive, multimodal artifact correction through attention mechanisms and specialized loss functions. The experimental protocols presented provide actionable methodologies for implementing these techniques across various research scenarios, from controlled behavioral tasks to fully naturalistic settings. As wearable neuroimaging continues to evolve toward real-world applications, robust artifact mitigation will remain crucial for validating data fusion techniques and extracting meaningful insights from brain dynamics in ecologically valid contexts.

Validating Fusion Performance: Metrics, Benchmarks, and Clinical Translation

In the realm of data fusion techniques for electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) research, selecting appropriate performance metrics is paramount for accurately evaluating classification models. EEG captures electrical brain activity with high temporal resolution but limited spatial resolution, while fNIRS measures hemodynamic responses with better spatial resolution but lower temporal resolution [5]. Their integration in a hybrid Brain-Computer Interface (BCI) aims to leverage these complementary characteristics to improve overall system performance [92] [93]. However, this multimodal approach introduces unique challenges for performance assessment, particularly given the typically imbalanced nature of brain signal datasets where target events (e.g., specific mental states) are often rare compared to baseline activity. This application note provides detailed protocols for using accuracy, AUC-ROC, and F1-score to evaluate classification performance within EEG-fNIRS fusion research, enabling researchers to select metrics that align with their specific experimental objectives and data characteristics.

Metric Definitions and Theoretical Foundations

Core Terminology

Binary classification metrics are derived from four fundamental outcomes organized in a confusion matrix [94] [95]:

• True Positive (TP): Correct prediction of the positive class
• True Negative (TN): Correct prediction of the negative class
• False Positive (FP): Incorrect positive prediction (actual is negative)
• False Negative (FN): Incorrect negative prediction (actual is positive)

Metric Formulas and Interpretation

Table 1: Fundamental Classification Metrics

Metric	Formula	Interpretation	Perfect Value
Accuracy	(TP + TN) / (TP + TN + FP + FN)	Overall correctness across both classes	1.0 (100%)
Precision	TP / (TP + FP)	Accuracy when predicting positive class	1.0
Recall (Sensitivity)	TP / (TP + FN)	Ability to identify all positive instances	1.0
F1-score	2 × (Precision × Recall) / (Precision + Recall)	Harmonic mean of precision and recall	1.0
False Positive Rate (FPR)	FP / (FP + TN)	Proportion of negatives incorrectly classified as positive	0.0

Metric Characteristics and Application Contexts

Accuracy

Accuracy provides an intuitive measure of overall correctness by measuring the proportion of all correct predictions among the total predictions made [94]. It is mathematically defined as:

[ \text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} ]

In Python, accuracy can be calculated using scikit-learn:

Application Context: Accuracy works well for balanced datasets where both classes are equally important and represented [96] [95]. However, in EEG-fNIRS research, accuracy becomes problematic for imbalanced datasets, which are common when detecting rare cognitive events or specific mental states. For instance, in a BCI where 95% of samples represent baseline activity and only 5% represent target mental commands, a model that always predicts the majority class would achieve 95% accuracy while being practically useless for detecting the target commands [94].

AUC-ROC

The Receiver Operating Characteristic (ROC) curve visualizes the trade-off between the True Positive Rate (TPR) and False Positive Rate (FPR) across all possible classification thresholds [96]. The Area Under the ROC Curve (AUC-ROC) provides a single number summarizing the model's ability to distinguish between classes, independent of any specific threshold.

Interpretation: AUC-ROC represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance [96]. A perfect model has an AUC of 1.0, while a random classifier has an AUC of 0.5.

Implementation:

Application Context: AUC-ROC is particularly valuable in EEG-fNIRS research when you care equally about both positive and negative classes and ultimately care about ranking predictions rather than outputting well-calibrated probabilities [96]. However, for heavily imbalanced datasets where the negative class dominates, the ROC curve can be overly optimistic because the false positive rate is pulled down due to the large number of true negatives [96].

F1-Score

The F1-score is the harmonic mean of precision and recall, providing a balanced metric that considers both false positives and false negatives [96] [94]. It is mathematically defined as:

[ \text{F1} = 2 \times \frac{\text{precision} \times \text{recall}}{\text{precision} + \recall} = \frac{2TP}{2TP + FP + FN} ]

Implementation:

Application Context: F1-score is the recommended metric for most EEG-fNIRS classification problems where you care more about the positive class [96]. It is especially valuable for imbalanced datasets common in BCI applications, such as detecting specific cognitive states or motor imagery tasks against background brain activity. The F1-score gives equal weight to precision and recall, making it suitable when both false positives and false negatives have consequences [94].

Table 2: Metric Selection Guide for EEG-fNIRS Research

Research Scenario	Recommended Metric	Rationale	EEG-fNIRS Example
Balanced dataset, equal importance of classes	Accuracy	Provides good overall assessment when classes are balanced	Subject-independent classification with equal trial distribution
Imbalanced dataset, focus on positive class	F1-score	Robust to class imbalance; balances precision and recall	Detection of rare epileptic spikes or specific cognitive states
Ranking predictions, overall separability	AUC-ROC	Measures ranking quality independent of threshold	Model selection during initial development phase
High cost of false positives	Precision	Emphasizes correctness of positive predictions	BCI communication systems where false commands are critical
High cost of false negatives	Recall	Emphasizes identifying all positive instances	Disease diagnosis or detection of critical neurological events

Experimental Protocols for EEG-fNIRS Classification

Protocol 1: Motor Imagery Classification with Hybrid EEG-fNIRS

Objective: To classify left-hand vs. right-hand motor imagery tasks using fused EEG-fNIRS features.

Dataset: Public dataset from Shin et al. (2017) containing 29 participants performing kinesthetic motor imagery (grasping a ball) with 30 trials per task [15].

Preprocessing Pipeline:

• EEG Processing: Downsample to 128Hz, apply common average reference, band-pass filter (8-25Hz) to retain μ-band and low-β band [15]
• fNIRS Processing: Convert raw light intensities to hemoglobin concentrations (HbO and HbR) using Modified Beer-Lambert Law
• Temporal Alignment: Ensure precise synchronization between EEG and fNIRS systems

Feature Extraction:

• EEG Features: Band power in μ-rhythm (8-13Hz) and β-rhythm (13-30Hz) from sensorimotor cortex channels
• fNIRS Features: Mean and slope of HbO signals during task period from motor cortex regions
• Feature Fusion: Concatenate normalized features from both modalities [92]

Classification & Evaluation:

• Apply 10×5-fold cross-validation with sliding window (window size: 3s, step: 1s)
• Train Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM) classifier
• Calculate accuracy, F1-score, and AUC-ROC for each fold
• Report mean ± standard deviation across all folds

Expected Performance: State-of-the-art methods achieve approximately 76% accuracy for bimodal classification, outperforming unimodal approaches (EEG-only: ~65%, fNIRS-only: ~57%) [15].

Protocol 2: Mental State Classification for BCI Applications

Objective: To distinguish between mental arithmetic (MA) and resting state using multimodal EEG-fNIRS.

Experimental Design:

• Tasks:
  • MA: Serial subtraction without verbalization or movement
  • Rest: Relaxed state with no specific mental activity
• Trial Structure: 2s instruction, 10s task period, 15-20s rest interval
• Participants: Minimum of 20 subjects for statistical power

Data Acquisition Specifications:

• EEG: 32+ channels including prefrontal, frontal, and parietal regions (sampling rate ≥200Hz)
• fNIRS: 16+ sources, 16+ detectors covering prefrontal cortex (sampling rate ≥10Hz)
• Synchronization: Hardware trigger or unified acquisition system for precise temporal alignment

Multimodal Fusion Strategies:

• Early Fusion: Combine raw or minimally processed data before feature extraction [15]
• Feature-Level Fusion: Extract features separately then concatenate (most common approach) [92]
• Decision-Level Fusion: Train separate classifiers then combine predictions [92]

Evaluation Framework:

• Use stratified k-fold cross-validation (k=5 or 10) to maintain class distribution
• Compute confusion matrix for each modality combination (EEG-only, fNIRS-only, fused)
• Calculate precision, recall, F1-score, accuracy, and AUC-ROC
• Perform statistical testing (e.g., paired t-test) to determine significant improvements from fusion

The Scientist's Toolkit: Essential Research Reagents and Materials

Table 3: Essential Research Materials for EEG-fNIRS Classification Experiments

Category	Item	Specification	Function/Purpose
Hardware Equipment	EEG Amplifier	BrainAmp, g.tec, or similar; ≥32 channels, ≥200Hz sampling	Records electrical brain activity with high temporal resolution
	fNIRS System	NIRScout, NIRx, or similar; ≥16 sources, ≥16 detectors	Measures hemodynamic responses via near-infrared light
	Integrated Cap	EEG electrodes + fNIRS optodes in unified assembly	Ensures precise co-registration and synchronization of modalities
Software Tools	Preprocessing	EEGLAB, NIRS-KIT, MNE-Python, Homer2	Signal filtering, artifact removal, and basic feature extraction
	Classification	scikit-learn, TensorFlow, PyTorch	Implementation of machine learning models and metrics
	Custom Analysis	MATLAB, Python with custom scripts	Specialized fusion algorithms and statistical analysis
Experimental Materials	Task Presentation	PsychoPy, Presentation, E-Prime	Controlled delivery of cognitive paradigms
	Data Synchronization	Lab Streaming Layer (LSL), trigger interface	Precise temporal alignment of multimodal data streams
Validation Resources	Public Datasets	Shin et al. (2017) motor imagery/mental arithmetic	Benchmarking and method comparison
	Performance Metrics	Accuracy, F1-score, AUC-ROC implementations	Comprehensive evaluation of classification performance

Advanced Considerations for Multimodal Fusion Research

Addressing Class Imbalance in EEG-fNIRS Data

In practical BCI applications, severe class imbalance is common, particularly in asynchronous BCIs where control states are rare compared to non-control states. Under these conditions:

• F1-score becomes increasingly important as it balances the trade-off between precision and recall [96]
• Precision-Recall (PR) curves may provide more meaningful assessment than ROC curves for imbalanced data [96]
• Strategic sampling techniques (oversampling minority class, undersampling majority class) can improve metric reliability
• Cost-sensitive learning approaches that assign higher misclassification costs to the minority class

Metric Selection Framework for Specific Applications

Clinical Diagnostic Applications (e.g., epilepsy detection, ADHD assessment):

• Primary metric: F1-score (balances false positives and false negatives)
• Secondary metrics: Sensitivity/Recall (minimize missed diagnoses)
• Threshold: Prioritize high sensitivity when disease consequences are severe

BCI Communication Applications:

• Primary metric: Accuracy (when balanced datasets) or F1-score (when imbalanced)
• Secondary consideration: Precision (minimize false commands)
• Real-time requirement: Metric stability across sessions

Neuromarketing/Cognitive Research:

• Primary metric: AUC-ROC (overall separability between conditions)
• Secondary metrics: Accuracy, F1-score
• Emphasis: Consistency across subjects rather than absolute performance

Reporting Standards for Publication

Comprehensive reporting should include:

• Multiple metrics (at minimum: accuracy, F1-score, AUC-ROC) to provide complete picture
• Cross-validation results (mean ± standard deviation)
• Statistical comparison between different fusion strategies
• Performance separated by modality (EEG-only, fNIRS-only, fused)
• Computational efficiency metrics when relevant for real-time applications

The selection of appropriate performance metrics is crucial for advancing EEG-fNIRS multimodal fusion research. While accuracy provides an intuitive overall measure, F1-score offers robustness against class imbalance common in BCI applications, and AUC-ROC assesses the overall ranking capability of classifiers. The experimental protocols and metric selection framework presented in this application note provide researchers with standardized methodologies for comprehensive evaluation of classification performance in multimodal brain signal research. By aligning metric selection with specific research objectives and data characteristics, the field can move toward more reproducible and comparable results across studies, ultimately accelerating progress in hybrid BCI development and clinical applications.

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a paradigm shift in neuroimaging research, offering a powerful solution to overcome the inherent limitations of single-modality approaches. Multimodal fusion leverages the complementary strengths of these two modalities: EEG provides millisecond-level temporal resolution of electrophysiological brain activity, while fNIRS offers superior spatial resolution for mapping hemodynamic responses [4] [6]. This synergy is particularly valuable for brain-computer interfaces (BCIs), cognitive state monitoring, and clinical diagnostics, where comprehensive neural decoding is essential.

The fundamental premise of EEG-fNIRS fusion stems from their complementary nature. EEG captures macroscopic cortical dynamics with fine temporal resolution (~5 msec) but suffers from low spatial resolution and sensitivity to artifacts [4]. Conversely, fNIRS measures hemodynamic properties with higher spatial resolution (~1 cm) and better resistance to motion artifacts, though it has significantly poorer temporal resolution [97] [98]. Through neurovascular coupling, these signals provide interrelated yet distinct perspectives on neural activity, creating a robust foundation for multimodal integration [99].

This Application Note provides a comprehensive quantitative comparison between single-modality and multimodal fusion approaches, detailing performance metrics, experimental protocols, and methodological considerations for researchers and drug development professionals working with neural signal analysis.

Quantitative Performance Comparison

Comprehensive Performance Metrics Across Methodologies

Table 1: Quantitative performance comparison of single-modality versus multimodal fusion approaches

Modality	Fusion Method	Task Domain	Classification Accuracy	Performance Improvement	Citation
EEG-only	ShallowConvNet	Motor Imagery (Multiple Joints)	65.49%	Baseline	[32]
fNIRS-only	Feature Selection	Visuo-Mental Task	~57%	Baseline	[15]
EEG-fNIRS	Mutual Information Feature Selection	Visuo-Mental Task	Significant improvement	~5% over single modalities	[4]
EEG-fNIRS	Early-Stage Fusion (Y-shaped Network)	Left/Right Motor Imagery	76.21%	~11% over EEG-only; ~19% over fNIRS-only	[15]
EEG-fNIRS	EFDA-CDG Framework	Mental Arithmetic	91.93%	Substantial improvement over single modalities	[98]
EEG-fNIRS	EFDA-CDG Framework	n-back Task	90.54%	Substantial improvement over single modalities	[98]
EEG-fNIRS	Multimodal MBC-ATT	Cognitive Tasks (n-back, WG)	Outperformed conventional approaches	Statistically significant	[6]
EEG-fNIRS	EFRM (Few-shot Learning)	Multiple Tasks	Competitive with state-of-the-art	High performance with minimal labeled data	[99]

Table 2: Advantages and limitations of fusion approaches

Fusion Category	Examples	Advantages	Limitations
Early Fusion	Feature concatenation, Y-shaped network [15]	Maximizes information sharing; Higher performance in motor imagery [15]	Susceptible to overfitting; Requires temporal alignment
Late Fusion	Classifier output combination, Fuzzy fusion [4]	Leverages modality-specific processing; More robust to noise	Potentially misses cross-modality interactions
Hybrid Methods	Mutual information feature selection [4], Cross-modal attention [6]	Optimizes complementarity; Reduces redundancy	Computationally intensive; Complex implementation

Key Performance Insights

Multimodal EEG-fNIRS fusion consistently demonstrates statistically significant improvements in classification accuracy across diverse task domains. The performance gains range from approximately 5% to over 19% compared to single-modality approaches, with the most substantial improvements observed in complex cognitive tasks [4] [15]. Early fusion strategies have shown particular promise for motor imagery tasks, achieving accuracy levels of 76.21% in left/right hand discrimination [15].

Advanced deep learning frameworks incorporating data augmentation and attention mechanisms have pushed performance boundaries further, achieving exceptional accuracy rates exceeding 90% for mental arithmetic and n-back tasks [98]. These approaches effectively address the challenge of limited training data through sophisticated augmentation techniques while leveraging the complementary information from both modalities.

Experimental Protocols & Methodologies

Mutual Information-Based Feature Selection Protocol

Objective: To classify fused EEG-fNIRS data by optimizing complementarity, redundancy, and relevance between multimodal features using mutual information metrics [4].

Equipment Setup:

• EEG system with 64-channel electrode cap following international 10-20 system
• fNIRS system with 8 sources and 8 detectors arranged in 8×8 grid
• Synchronized data acquisition hardware

Experimental Procedure:

• Participant Preparation (30 minutes)
  • Recruit subjects (e.g., 9 ALS patients vs. 9 healthy controls)
  • Apply EEG cap with electrode impedance maintained below 10 kΩ
  • Position fNIRS optodes on same recording cap covering motor cortex regions
  • Verify signal quality from both modalities

• Data Acquisition (60-90 minutes)

  • Conduct visuo-mental task paradigm with randomized trial sequence
  • Simultaneously record EEG (1000 Hz sampling) and fNIRS (7.8125 Hz sampling)
  • Implement 50 Hz notch filter for power line interference removal
  • Apply band-pass filtering (EEG: 0.5-100 Hz; fNIRS: 0.01-0.1 Hz)

• Signal Processing Pipeline

  • Preprocess raw signals: artifact removal using ICA, wavelet denoising
  • Extract temporal and spectral features from both modalities
  • Extract HbO and HbR concentration changes from fNIRS data

• Mutual Information Feature Selection

  • Compute mutual information between features and class labels
  • Select feature subset maximizing relevance to classes while minimizing redundancy
  • Apply cross-validation to optimize feature subset selection

• Classification & Validation

  • Train classifier (e.g., SVM, LDA) on selected feature subset
  • Evaluate performance using cross-validation
  • Compare against single-modality benchmarks

Early-Stage Fusion Y-Shaped Network Protocol

Objective: To investigate fusion stages and determine optimal integration point for EEG-fNIRS signals in motor imagery classification [15].

Dataset:

• Use publicly available Dataset A from Shin et al. (29 participants)
• Left-hand vs. right-hand motor imagery tasks
• 30 trials per task per participant

Processing Workflow:

Diagram 1: Early fusion experimental workflow

EEG Processing Branch:

• Downsample from 200 Hz to 128 Hz
• Remove EOG channels and re-reference to common average
• Apply 8-25 Hz band-pass filter (μ-band and low-β band)
• Extract spatial-temporal features using modified EEGNet architecture

fNIRS Processing Branch:

• Convert raw light intensities to HbO and HbR concentrations
• Apply 0.01-0.2 Hz band-pass filter to remove physiological noise
• Extract temporal features using second and third modules of EEGNet

Fusion & Classification:

• Implement early fusion by concatenating features from both modalities before fully connected layers
• Compare with middle and late fusion strategies
• Train network using leave-one-out cross-validation
• Evaluate classification accuracy for left vs. right motor imagery

Cross-Modal Attention Fusion Protocol (MBC-ATT)

Objective: To implement cross-modal attention mechanism for selectively integrating EEG and fNIRS signals in cognitive state decoding [6].

Dataset Preparation:

• Utilize open-access multimodal dataset with simultaneous EEG-fNIRS recordings
• Include n-back and word generation tasks from 26 healthy participants
• Ensure proper trial structure with randomized task presentation

Network Architecture:

• Independent Modality Branches
  • Process EEG and fNIRS through separate convolutional neural networks
  • Extract temporal features from EEG using 1D convolutions
  • Extract spatial-temporal patterns from fNIRS using 2D convolutions

• Cross-Modal Attention Module

  • Compute attention weights between modality representations
  • Implement modality-guided attention to emphasize complementary features
  • Dynamically adjust feature contributions based on task relevance

• Fusion & Classification

  • Integrate attended features through weighted combination
  • Pass through fully connected layers for final classification
  • Compare with conventional fusion approaches (concatenation, averaging)

Validation Method:

• Perform subject-independent cross-validation
• Compare with single-modality baselines and state-of-the-art methods
• Conduct ablation studies to validate attention mechanism contribution

Signaling Pathways & Neurophysiological Basis

The effectiveness of EEG-fNIRS multimodal fusion is grounded in the fundamental principles of neurovascular coupling, the biological process that links neural activity to subsequent hemodynamic responses [99].

Diagram 2: Neurovascular coupling in multimodal fusion

Temporal Relationships:

• EEG Response: Immediate onset (milliseconds post-stimulus), reflecting direct neuronal firing and synchronization [6]
• fNIRS Response: Delayed onset (2-6 seconds post-stimulus), peaking at 5-8 seconds, reflecting hemodynamic changes [97]
• Complementary Information: EEG captures initial neural activation, while fNIRS provides sustained activity monitoring

Neurophysiological Basis:

• Neural Activation triggers increased metabolic demand for oxygen and glucose
• Neurovascular Coupling mechanisms initiate localized blood flow changes
• Hemodynamic Response manifests as increased HbO and decreased HbR concentrations
• Spatial-Temporal Complementarity: EEG offers millisecond temporal precision while fNIRS provides centimeter-level spatial specificity [97] [99]

This biological relationship creates the foundation for effective multimodal fusion, where electrophysiological and hemodynamic measures provide complementary views of the same underlying neural processes.

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential equipment and computational tools for EEG-fNIRS multimodal research

Category	Specific Tool/Solution	Function/Purpose	Example Specifications
Acquisition Hardware	EEG Cap & Amplifier	Records electrical brain activity	64-channel setup; 1000 Hz sampling; <10 kΩ impedance [32]
Acquisition Hardware	fNIRS System with Optodes	Measures hemodynamic responses	8 sources, 8 detectors; 7.8125 Hz sampling; 24 channels [32]
Software & Analysis	Mutual Information Toolkits	Feature selection and optimization	Maximizes complementarity, minimizes redundancy [4]
Software & Analysis	Deep Learning Frameworks	Implements fusion architectures	PyTorch/TensorFlow with custom Y-shaped networks [15]
Software & Analysis	Data Augmentation Tools	Expands limited training datasets	Denoising Diffusion Probabilistic Models (DDPM) [98]
Experimental Materials	Multimodal Recording Caps	Integrated EEG electrodes & fNIRS optodes	Custom designs with 10-20 system compatibility [32]
Experimental Materials	Signal Quality Verification Kits	Ensures data integrity during acquisition	Impedance checkers, light intensity monitors [32]

Multimodal EEG-fNIRS fusion represents a significant advancement over single-modality approaches, consistently demonstrating superior performance across diverse applications from motor imagery to cognitive state assessment. The quantitative evidence confirms that properly implemented fusion strategies can achieve performance improvements of 5-19% over individual modalities, with advanced frameworks reaching exceptional accuracy levels exceeding 90% for specific tasks [4] [98] [15].

The optimal fusion strategy depends on the specific application requirements: early fusion approaches excel in motor imagery tasks, while attention-based mechanisms show promise for complex cognitive state decoding. Future directions include standardized preprocessing pipelines, transfer learning approaches for limited data scenarios, and real-time implementation for clinical and BCI applications [98] [99]. As multimodal methodologies continue to mature, they hold tremendous potential to transform neuroimaging research and clinical brain monitoring applications.

Public Datasets and Benchmarking Standards for Method Evaluation

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a promising direction for brain activity decoding with high spatiotemporal resolution in naturalistic scenarios [7]. However, the advancement of data fusion techniques for these multimodal signals faces significant challenges due to the lack of standardized public datasets and robust benchmarking frameworks. This creates substantial variability in how machine learning is applied to fNIRS data and how methodology and results are reported, making it difficult to evaluate whether some approaches are genuinely superior to others [100]. This application note provides a comprehensive resource for researchers seeking to navigate the current landscape of public datasets, benchmarking standards, and experimental protocols for EEG-fNIRS fusion research, with particular emphasis on their application within therapeutic development contexts.

Available Public Datasets for EEG-fNIRS Research

The emergence of publicly available, multimodal datasets is crucial for developing and validating data fusion algorithms. These datasets enable direct comparison of methods and facilitate reproducible research. Several recent datasets have been curated to support specific research applications and are summarized in Table 1.

Table 1: Publicly Available Multimodal EEG-fNIRS Datasets

Dataset Name	Primary Research Focus	Subjects (Healthy/Patients)	Recorded Tasks	Key Modalities	Data Format & Access
HEFMI-ICH [31]	Intracerebral Hemorrhage Rehabilitation	17 Normal / 20 ICH Patients	Left/Right Hand Motor Imagery	32-ch EEG, 90-ch fNIRS (HbO, HbR)	Raw & pre-processed trial data; Clinical scores (FMA-UE, MBI)
Multimodal Upper-Limb Joint MI [32]	Fine Motor Imagery Decoding	18 Healthy	8 MI tasks for Hand, Wrist, Elbow, Shoulder	64-ch EEG, 24-ch fNIRS	Raw recordings; 5760 total trials
REFED [101]	Affective Computing (aBCI)	32 Healthy	Emotion evocation via video	EEG, fNIRS (HbO, HbR, HbT)	Raw data; Real-time valence/arousal annotations
Shin et al. Dataset A [15]	Basic Motor Imagery Benchmarking	29 Healthy	Left/Right Hand Motor Imagery	EEG, fNIRS (HbO)	Pre-processed; Standard left/right MI
Semantic Imagery Dataset [81]	Semantic Neural Decoding	12 Healthy (+7 for EEG-only)	Silent Naming, Visual, Auditory, Tactile Imagery of Animals/Tools	Simultaneous EEG-fNIRS	Raw data; Semantic category labels

These datasets support various research applications, from motor rehabilitation to affective computing. The HEFMI-ICH dataset is particularly notable for including patient data, addressing a critical gap in clinical translation [31]. For drug development professionals, such patient-derived datasets provide invaluable resources for evaluating neurotherapeutics targeting motor recovery or cognitive function.

Benchmarking Frameworks and Standards

To address the lack of community standards for applying machine learning to fNIRS data, the BenchNIRS framework has been developed as an open-source benchmarking tool [100]. This framework establishes a best-practice methodology using nested cross-validation to optimize models and evaluate them without bias, providing standardized metrics and figures for performance comparison.

Performance benchmarking across multiple datasets reveals that actual classification accuracy is often lower than scores frequently reported in literature, with minimal differences between model types when evaluated rigorously [100]. This underscores the importance of standardized benchmarking to prevent overestimation of model capabilities, particularly when transitioning from research to clinical applications.

Experimental Protocols and Methodologies

Standardized Motor Imagery Protocol

A common experimental paradigm for EEG-fNIRS research involves motor imagery tasks with standardized timing. A typical protocol follows this structure [31]:

• Baseline Recording (2 minutes): Eyes-closed (1 minute) followed by eyes-open (1 minute) resting state, demarcated by an auditory cue.
• Trial Structure:
  • Visual Cue (2 seconds): Presentation of a directional arrow indicating the required motor imagery task.
  • Execution Phase (10 seconds): Participants perform kinesthetic motor imagery (e.g., imagining grasping a ball at ~1 repetition/second) while focusing on a central fixation cross.
  • Inter-Trial Interval (15 seconds): Blank screen for rest, allowing hemodynamic responses to return to baseline.
• Session Design: Multiple sessions with ≥30 trials each, with adequate rest intervals between sessions to mitigate fatigue.

This protocol is implemented in the HEFMI-ICH dataset and can be adapted for various patient populations [31]. For pharmacological studies, the baseline and post-intervention assessments can be incorporated within this framework to evaluate drug effects on motor imagery-related brain activity.

Data Acquisition and Preprocessing Standards

Simultaneous EEG-fNIRS acquisition requires careful hardware integration. A standard setup includes [31]:

• EEG System: 32+ channels with sampling rate ≥256 Hz, positioned according to the international 10-20 system.
• fNIRS System: Continuous-wave system with multiple sources and detectors (e.g., 32 sources, 30 detectors) arranged to provide coverage of target cortical areas (e.g., sensorimotor cortex), generating ~90 measurement channels through source-detector pairing at ~3 cm separation.
• Synchronization: Event markers from stimulus presentation software (e.g., E-Prime) simultaneously trigger both recording systems.

A standardized preprocessing pipeline is essential for reproducible results. The workflow below outlines the critical steps for preparing raw EEG and fNIRS signals for subsequent analysis or fusion experiments.

Data Fusion Methodologies

Multiple fusion strategies have been developed to integrate EEG and fNIRS signals, each with distinct advantages. Research comparing these approaches has demonstrated that early-stage fusion significantly outperforms middle-stage and late-stage fusion for classifying motor imagery tasks [15]. The diagram below illustrates the architectural differences and information flow between these three primary fusion strategies.

The Scientist's Toolkit: Essential Research Reagents

Successful execution of EEG-fNIRS experiments requires specific hardware, software, and analytical tools. Table 2 catalogues the essential "research reagents" for multimodal brain signal research.

Table 2: Essential Research Reagents for EEG-fNIRS Experiments

Category	Item	Specification/Example	Primary Function
Hardware	EEG Amplifier	g.HIamp (g.tec), Neuroscan SynAmps2	High-fidelity electrical signal acquisition (≥256 Hz)
	fNIRS System	NirScan (Danyang Huichuang), NIRScout (NIRx)	Hemodynamic response measurement via near-infrared light
	Hybrid Cap	Custom designs with integrated EEG electrodes & fNIRS optodes	Simultaneous signal acquisition with precise spatial co-registration
Software	Stimulus Presentation	E-Prime, PsychoPy, Presentation	Precise experimental paradigm control and event marker generation
	Data Analysis	EEGLAB, FieldTrip, MNE-Python, HomER2, NIRS-KIT	Signal processing, visualization, and statistical analysis
	Machine Learning	BenchNIRS [100], Scikit-learn, PyTorch, TensorFlow	Model development, training, and benchmarking
Analytical Components	Clinical Scales	Fugl-Meyer Assessment (FMA-UE), Modified Barthel Index (MBI)	Quantitative assessment of motor function and independence
	Self-Assessment Scales	SAM (Valence/Arousal), PANAS	Subjective emotion and affective state measurement

The developing ecosystem of public datasets and benchmarking standards for EEG-fNIRS research provides critical infrastructure for advancing data fusion techniques. The available resources support diverse applications from motor rehabilitation to affective computing, while emerging frameworks like BenchNIRS address the crucial need for standardized evaluation. For researchers in academia and drug development, these tools enable more reproducible, comparable, and clinically relevant studies of brain function and therapeutic interventions. Future work should focus on expanding patient-specific datasets, validating fusion algorithms in clinical populations, and establishing consensus standards for reporting results across the research community.

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a transformative approach in clinical neuroscience, enabling a more comprehensive investigation of brain function by capturing complementary neural signatures. EEG provides millisecond-level temporal resolution of electrophysiological activity, while fNIRS tracks hemodynamic responses with superior spatial localization [5]. This multimodal data fusion framework offers a powerful tool for identifying robust biomarkers and elucidating the underlying neuropathophysiology of neurological and psychiatric disorders, ultimately advancing objective diagnostic and therapeutic monitoring capabilities [4] [5].

This application note details experimental protocols and presents clinical validation case studies for three distinct conditions: Parkinson's disease, Attention-Deficit/Hyperactivity Disorder, and Major Depressive Disorder. The synergistic combination of EEG and fNIRS leverages complementary information; EEG reflects neuronal electrical discharges, whereas fNIRS measures hemodynamic changes associated with regional neural metabolism [5]. This dual-modality approach is particularly valuable for investigating complex phenomena such as cognitive-motor interference and neurovascular coupling, which are often impaired in various brain disorders [102].

The Scientist's Toolkit: Essential Research Reagents & Materials

Successful execution of simultaneous EEG-fNIRS studies requires specific hardware, software, and methodological components. The table below catalogues the essential solutions for integrated system configuration, signal acquisition, and data processing.

Table 1: Key Research Reagent Solutions for Integrated EEG-fNIRS Studies

Item Name	Function/Application	Technical Notes
Integrated EEG-fNIRS Helmet	Co-registration of electrophysiological and hemodynamic signals from homologous brain regions.	Custom designs via 3D printing or cryogenic thermoplastic sheets improve scalp coupling and placement consistency [5].
Synchronization Module	Temporal alignment of EEG and fNIRS data streams with high precision.	A unified processor for simultaneous acquisition is preferred over post-hoc software synchronization [5].
fNIRS Probe Configurations	Measures concentration changes in oxygenated (HbO) and deoxygenated (HbR) hemoglobin.	Source-detector distances of 3 cm are typical to ensure adequate cortical penetration [5].
EEG Recording System	Records electrical potentials from the scalp with high temporal resolution.	A 32-channel system following the 10-20 international standard is common for full-brain coverage [103].
Task-Related Component Analysis (TRCA)	A computational algorithm to extract task-reproducible components from bimodal signals, enhancing signal-to-noise ratio.	Effectively improves the reproducibility and discriminability of neural patterns for subsequent analysis [102].
Mutual Information-Based Feature Selection	A filter method for selecting an optimized subset of features from the fused EEG-fNIRS dataset by maximizing relevance and minimizing redundancy.	Helps improve hybrid classification performance by exploiting feature complementarity with respect to class labels [4].
Correlation-Based Signal Improvement (CBSI)	A combined hemoglobin measure used in fNIRS analysis to mitigate motion artifacts and improve signal quality.	Validated as a robust measure of executive demand during walking tasks in Parkinson's disease [104].

Clinical Case Studies & Experimental Protocols

Case Study: Cognitive-Motor Interference in Parkinson's Disease

Background & Validation Objective: Parkinson's disease (PD) is characterized by motor symptoms and cognitive decline. Cognitive-motor interference (CMI) occurs when simultaneous performance of a motor and cognitive task leads to deterioration in one or both, a common challenge in PD that increases fall risk [102] [104]. This study validated fNIRS measurement of executive demand during walking with and without a dual-task in younger adults, older adults, and people with PD [104].

Experimental Protocol:

• Participants: Recruit 42 participants from three groups: younger adults (18-50 years), older adults (≥60 years), and people with PD (≥60 years) [104].
• Task Paradigm:
  • Single-Task Walking (ST): Participants walk at a self-selected pace.
  • Dual-Task Walking (DT): Participants walk while concurrently performing an auditory Stroop task (e.g., responding to the meaning of a word while ignoring the pitch of the voice) [104].
• Data Acquisition:
  • fNIRS: Place probes over the prefrontal cortex, focusing on the dorsolateral prefrontal cortex (dlPFC). Calculate the Correlation-Based Signal Improvement (CBSI) as a combined hemoglobin measure [104].
  • Kinematics: Record gait parameters (e.g., walking speed, step time variability) using motion capture or inertial measurement units.
• Validation Hypotheses:
  • Convergent Validity: Pre-register hypotheses that dlPFC activity (CBSI) during DT walking positively correlates with dual-task cost (the performance decline on one task from single to dual-task) [104].
  • Known-Group Validity: Hypothesize that the increase in dlPFC activity from ST to DT will differ between participant groups.

Key Findings & Clinical Relevance: The fNIRS CBSI measure was validated as an indicator of executive demand during walking. A positive relationship was found between dlPFC activity during dual-task walking and dual-task cost in younger adults. In the PD group, dlPFC activity during single-task walking was positively related to step time variability and negatively related to walking speed [104]. This protocol provides an objective method to quantify the cognitive burden of walking in PD, which can be used to assess disease progression and the efficacy of rehabilitative interventions.

Case Study: Hybrid Classification for Major Depressive Disorder (MDD)

Background & Validation Objective: Major Depressive Disorder diagnosis often relies on subjective clinical interviews and rating scales, leading to a need for objective biomarkers. This study aimed to establish an automated diagnostic assessment for MDD by combining EEG and fNIRS to improve classification accuracy [103].

Experimental Protocol:

• Participants: Recruit 25 patients with MDD and 30 demographically matched healthy controls (HCs). Patient diagnosis must conform to DSM-V criteria, confirmed by two attending psychiatrists [103].
• Task Paradigm: A 6-minute resting-state paradigm with eyes closed is employed. Participants sit comfortably in a quiet room and are instructed to remain awake and relaxed [103].
• Data Acquisition:
  • EEG: Record from 28 channels according to the international 10-20 system. Calculate brain functional network properties (e.g., clustering coefficient, local efficiency) in delta, theta, and alpha bands. Compute hemispheric asymmetry in the theta band [103].
  • fNIRS: Record from the forehead (prefrontal cortex). Extract hemodynamic features, including brain oxygen sample entropy [103].
• Data Fusion & Analysis:
  • Feature Extraction: Extract candidate features from both modalities, including EEG brain network properties, asymmetry, and fNIRS-based entropy.
  • Feature Selection: Implement a data-driven automated feature selection method to identify the most discriminative features.
  • Classification: Train a Support Vector Machine (SVM) model with selected features to classify MDD vs. HC [103].

Key Findings & Clinical Relevance: The hybrid EEG-fNIRS model achieved a classification accuracy of 92.7%, significantly outperforming the model using EEG features alone (81.8%) [103]. Key biomarkers included brain network local efficiency in the delta band, hemispheric asymmetry in the theta band, and brain oxygen sample entropy. This demonstrates the substantial potential of multimodal data fusion in providing an objective, physiological-based diagnostic tool for depression.

Case Study: Unveiling Neural Mechanisms in Attention-Deficit/Hyperactivity Disorder (ADHD)

Background & Validation Objective: ADHD is a heterogeneous neurodevelopmental disorder. Machine learning (ML) applied to multimodal data can help unravel its complex neural mechanisms beyond what is possible with single-modality studies [105].

Experimental Protocol:

• Participants: Include children with ADHD and typically developing controls, with sample sizes large enough for robust machine learning (ideally >40 per group) [105].
• Task Paradigm: Administer tasks probing core cognitive deficits in ADHD, such as response inhibition (e.g., Go/No-Go task), sustained attention (e.g., Continuous Performance Task), and working memory [105].
• Data Acquisition:
  • EEG: Record event-related potentials (ERPs) like P300 and measure spectral power (e.g., theta/beta ratio) during cognitive tasks.
  • fNIRS: Record hemodynamic responses from prefrontal and parietal cortices during task performance to assess regional brain activation.
• Data Fusion & Analysis:
  • Feature Engineering: Extract temporal and spectral features from EEG and hemodynamic features (HbO, HbR) from fNIRS.
  • Machine Learning: Use algorithms like Support Vector Machines (SVM) or Random Forest. Perform feature selection to identify the most critical neurophysiological and hemodynamic features contributing to classification. The goal is not just classification but also understanding the disorder's mechanisms by interpreting these features [105].

Key Findings & Clinical Relevance: Machine learning models that integrate multimodal data have successfully identified that core characteristics of ADHD extend beyond traditional symptom descriptions. For instance, models have highlighted the discriminative power of teachers' ratings on oppositional questions and the significance of factors like family history and academic difficulties [105]. The application of ML to EEG and fNIRS data holds promise for discovering objective biomarkers that can aid in subtyping ADHD and predicting treatment response, moving towards precision medicine.

The following table synthesizes key performance metrics and findings from the featured case studies and related research, highlighting the efficacy of the EEG-fNIRS fusion approach.

Table 2: Summary of Quantitative Results from EEG-fNIRS Clinical Validation Studies

Clinical Condition	Primary Metric	EEG Alone	fNIRS Alone	Fused EEG-fNIRS	Key Biomarkers Identified
Major Depressive Disorder (MDD) [103]	Classification Accuracy	81.8%	Information Missing	92.7%	Delta band local efficiency, Theta band asymmetry, Brain oxygen sample entropy
Psychological Stress [4] [103]	Classification Accuracy	~3.4% improvement vs. single modality	~11% improvement vs. single modality	~5-11% increase vs. best single modality	Mutual information-based selected features
Mental Workload [103]	Classification Accuracy	85.9%	74.8%	90.9%	Hybrid temporal-spectral-hemodynamic features
Amyotrophic Lateral Sclerosis (ALS) [4]	Hybrid Classification Performance	Information Missing	Information Missing	Significantly improved vs. single modality and conventional classification	Optimized feature subset from mutual information selection

Workflow & Signaling Diagrams

Multimodal Experimental Fusion Workflow

The diagram below illustrates the end-to-end pipeline for acquiring, processing, and analyzing simultaneous EEG-fNIRS data in clinical validation studies.

Neurovascular Coupling in Cognitive-Motor Interference

This diagram conceptualizes the neural processes and their measurement during a dual-task paradigm, relevant to Parkinson's disease and other conditions affecting cognitive-motor function.

Application Notes

Motor Imagery (MI) decoding is a cornerstone of modern Brain-Computer Interface (BCI) systems, enabling direct communication between the brain and external devices by interpreting the neural activity associated with imagined movements without any physical execution. This technology has transformative potential, particularly in neurorehabilitation, assisting patients with motor disabilities, stroke, ALS, or spinal cord injuries in regaining communication and control capabilities [106] [107].

The core challenge in MI-BCI systems lies in accurately classifying intention from noisy, non-stationary Electroencephalogram (EEG) signals. Recent breakthroughs have significantly enhanced performance through two key paradigms: the integration of source localization techniques with deep learning, and the application of sophisticated domain adaptation frameworks to overcome the variability in brain signals across different individuals [108] [109].

Table 1: Quantitative Performance of Advanced MI Decoding Models

Model / Approach	Core Methodology	Dataset(s)	Performance (Accuracy)	Key Advantage
Beamforming + ResNet CNN [108]	Source localization (Beamforming) transformed to cortical maps, analyzed by a custom CNN.	4-class MI (Left hand, Right hand, Both feet, Tongue)	99.15% (within-subject)	Superior spatial resolution; outperforms sensor-domain approaches.
TFANet [110]	Temporal Fusion Attention Network with Multi-Scale Temporal Self-Attention (MSTSA).	BCIC-IV-2a, BCIC-IV-2b	84.92% (2a), 88.41% (2b) (within-subject)	Effectively captures complex temporal dependencies in EEG signals.
MSDCDA [109]	Multi-source Dynamic Conditional Domain Adaptation Network with dynamic residual modules.	BCIC-IV-2a, BCIC-IV-2b	78.55% (2a), 85.08% (2b) (cross-subject)	Mitigates inter-subject variability; enables effective cross-user decoding.

The performance gains from these models are a substantial improvement over traditional machine learning methods, which often rely on hand-crafted features and struggle with signal noise and subject-specific variations [110]. Furthermore, the fusion of EEG with other modalities like functional near-infrared spectroscopy (fNIRS) represents a promising direction. This multimodal approach compensates for the inherent limitations of each technique—combining the excellent temporal resolution of EEG with the superior spatial resolution and metabolic information of fNIRS—paving the way for more robust and naturalistic brain imaging systems [7].

Experimental Protocols

Protocol 1: Source Localization with Beamforming and Deep Learning

This protocol details the methodology for achieving state-of-the-art within-subject classification accuracy [108].

A. Data Acquisition & Preprocessing

• Paradigm: Subjects perform cued MI tasks (e.g., left hand, right hand, both feet, tongue). Each trial is timed, starting with a fixation cross, followed by a visual cue indicating the task, and an MI period.
• EEG Recording: Data is collected using a multi-channel EEG system (e.g., 22 channels) at a sampling rate of 250 Hz.
• Epoching: EEG signals are segmented into trials relative to the cue onset (e.g., a 4-second window from 0 to 4 seconds after the cue).
• Filtering: Apply a band-pass filter (e.g., 8-30 Hz) to isolate the μ and β rhythms associated with sensorimotor activity.

B. Source Localization via Beamforming

• The preprocessed sensor-level EEG signals are projected onto cortical source space.
• Beamforming is used to solve the inverse problem and estimate the current density of neuronal sources on a cortical surface. This transforms the multi-channel EEG time-series into a series of 2D cortical activity maps.

C. Deep Learning Model Training & Classification

• Architecture: A custom Residual Network (ResNet) Convolutional Neural Network (CNN) is designed to process the cortical activity maps.
• Input: The sequence of 2D source activity maps for each trial.
• Training: The model is trained in a supervised manner using the labeled source-localized trials to classify the type of motor imagery.
• Validation: Performance is evaluated on a held-out test session from the same subject.

Protocol 2: Cross-Subject Decoding with Domain Adaptation

This protocol addresses the challenge of applying a model trained on one subject to a new, unseen subject, a critical requirement for real-world BCI deployment [109].

A. Multi-Source Data Preparation

• Source Domains: Gather labeled EEG data from multiple subjects (e.g., 8 subjects from a public dataset like BCIC-IV-2a).
• Target Domain: Use the unlabeled EEG data from a new target subject. The data from all subjects is preprocessed similarly (e.g., band-pass filtering, epoching).
• Feature Extraction: A shared feature extractor network, often incorporating spatial-temporal convolution blocks and attention mechanisms, is used to generate features from all subjects.

B. Dynamic Domain Adaptation

• Dynamic Residual Module: This module dynamically adjusts network parameters based on the input sample, allowing the model to adapt to the unique feature distribution of each subject and alleviating conflicts between multiple source domains.
• Multi-Channel Attention Block: Focuses the model on EEG channels that are most relevant to the MI task, improving feature discriminability.
• Adversarial Alignment with MDD: The Margin Disparity Discrepancy (MDD) metric is used with an auxiliary classifier to perform adversarial training. This aligns the conditional distributions (i.e., the distributions per class) of the source and target domains, ensuring that features for the same MI task are similar across different subjects.

C. Model Evaluation

• The classifier, trained on labeled source data and adapted to the target data, is used to predict labels for the target subject's trials.
• Accuracy is calculated by comparing these predictions to the ground-truth labels (which are used for evaluation only).

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Materials and Computational Tools for MI-Decoding Research

Item	Function / Application in MI Research
High-Density EEG System	Records electrical brain activity from the scalp with high temporal resolution. Essential for capturing event-related desynchronization (ERD) during motor imagery [109].
fNIRS/HD-DOT System	Measures hemodynamic changes (blood oxygenation) in the cortex. Used in multimodal fusion with EEG to provide complementary spatial information and validate neuronal activity [7].
Public BCI Datasets (e.g., BCIC-IV-2a/2b)	Provide standardized, annotated EEG data for benchmarking and training new decoding algorithms, ensuring reproducibility and comparability of results [110].
Source Localization Algorithms (Beamforming, MNE)	Computational methods to solve the EEG inverse problem, projecting sensor-level signals to their origins on the cortical surface, thereby enhancing spatial information [108].
Domain Adaptation Frameworks (e.g., MSDCDA)	Software and model architectures designed to mitigate the performance drop caused by inter-subject variability, a major hurdle for practical BCI systems [109].
Synthetic fNIRS-EEG Datasets	Computer-generated datasets that simulate realistic, co-registered multimodal signals. They are crucial for developing and testing fusion algorithms in the absence of large, real-world multimodal datasets [7].

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) represents a paradigm shift in neuroimaging, enabling unprecedented insights into brain function through complementary data streams. This multimodal approach leverages EEG's excellent temporal resolution with fNIRS's superior spatial specificity to overcome the limitations of individual modalities [5]. The convergence of these technologies creates a powerful framework for cross-disciplinary validation, bridging the gap between basic neuroscience discoveries and clinical applications in psychiatry and neurology [111]. This protocol details standardized methodologies for fNIRS-EEG data fusion, with particular emphasis on translational applications in therapeutic development and clinical diagnostics.

The fundamental rationale for EEG-fNIRS integration stems from their complementary nature: EEG captures millisecond-scale neuronal electrical activity, while fNIRS measures hemodynamic responses with spatial precision superior to EEG [5]. This combination provides a more complete picture of brain activity by simultaneously tracking electrophysiological and neurovascular components [7]. Furthermore, both modalities share practical advantages including non-invasiveness, portability, and relatively low cost compared to fMRI and PET, making them ideal for naturalistic experimental settings and diverse patient populations [5].

Technical Foundations of EEG-fNIRS Integration

Core Physiological Principles and Signal Characteristics

Table 1: Technical Specifications and Comparative Analysis of Neuroimaging Modalities

Technique	Temporal Resolution	Spatial Resolution	Invasiveness	Key Measured Parameters	Primary Applications
EEG	Millisecond-scale (excellent)	~1-2 cm (low)	Non-invasive	Electrical potentials from synchronized neuronal firing	Epilepsy monitoring, sleep studies, brain-computer interfaces
fNIRS	Seconds (moderate)	~1-2 cm (moderate)	Non-invasive	Concentration changes in oxygenated (HbO) and deoxygenated hemoglobin (HbR)	Functional brain mapping, cognitive neuroscience, clinical monitoring
fMRI	1-2 seconds (moderate)	1-3 mm (high)	Non-invasive	Blood oxygenation level dependent (BOLD) signal	Pre-surgical mapping, cognitive task localization, connectomics
PET	Minutes (poor)	3-5 mm (high)	Injective (radioactive tracers)	Metabolic activity, receptor binding, neurotransmitter dynamics	Oncology, neuropharmacology, metabolic studies
MEG	Millisecond-scale (excellent)	3-5 mm (high)	Non-invasive	Magnetic fields generated by neuronal activity	Source localization, epilepsy focus identification

The synergistic potential of EEG and fNIRS emerges directly from their complementary technical profiles. EEG's exceptional temporal resolution (milliseconds) captures neural dynamics with precision unmatched by hemodynamic methods, while fNIRS provides spatial localization superior to EEG alone [5]. Crucially, both modalities operate without electromagnetic interference, enabling truly simultaneous data acquisition – a significant advantage over combinations involving fMRI or MEG [5].

System Integration Architectures

Two primary approaches exist for hardware integration of EEG-fNIRS systems:

• Synchronized Separate Systems: EEG and fNIRS data are acquired using independent equipment (e.g., NIRScout and BrainAMP systems) with post-hoc synchronization during analysis [5]. While simpler to implement, this approach may suffer from precision limitations for microsecond-scale EEG analysis.

• Unified Processor Systems: A single integrated unit processes and acquires both EEG and fNIRS signals simultaneously [5]. This architecture provides superior synchronization precision and streamlines analytical workflows, making it the preferred approach despite greater system complexity.

Helmet design represents a critical integration challenge. Current solutions include:

• Integrated substrate materials with EEG electrodes and fNIRS probes on a shared platform
• Separate arrangement of EEG electrodes and NIR fiber-optic components
• Modified EEG caps with punctures for fNIRS probe fixtures
• Customized helmets using 3D printing or cryogenic thermoplastic sheets for improved fit and probe stability [5]

Data Fusion Methodologies and Experimental Protocols

Multimodal Fusion Strategies

Table 2: Data Fusion Techniques for EEG-fNIRS Integration

Fusion Level	Description	Common Algorithms/Methods	Advantages	Limitations
Early Fusion (Data-level)	Raw or preprocessed data combined before feature extraction	Y-shaped neural networks [15]	Preserves temporal relationship; enables shared representation learning	Susceptible to noise propagation; requires temporal alignment
Middle Fusion (Feature-level)	Features extracted separately then combined	Common Spatial Pattern (CSP) + hemoglobin mean/slope [15], Canonical Correlation Analysis [15]	Leverages modality-specific processing; more robust to noise	May miss cross-modal interactions; requires careful feature selection
Late Fusion (Decision-level)	Separate classification with combined decisions	Weighted voting, Bayesian fusion	Flexible to modality-specific classifiers; robust to missing data	Loses low-level interactions; may yield suboptimal performance

Evidence suggests that early-stage fusion of EEG and fNIRS significantly outperforms middle and late fusion approaches in classification tasks. In motor imagery experiments, Y-shaped neural networks implementing early fusion demonstrated statistically higher performance (N = 57, P < 0.05), achieving average accuracy of 76.21% in left/right hand discrimination tasks [15].

Standardized Experimental Protocol for Motor Imagery Paradigm

Protocol 1: Simultaneous EEG-fNIRS Acquisition for Motor Imagery Classification

This protocol details the experimental setup based on Dataset A from Shin et al. (2017), which has been widely used in hybrid BCI research [15].

I. Equipment Setup

• EEG system: BrainAmp EEG amplifier with sampling rate ≥200 Hz
• fNIRS system: Compatible continuous-wave system with appropriate optode configuration
• Integrated cap: 3D-printed or thermoplastic custom helmet with 20 EEG electrodes (following 10-20 system) and 16 fNIRS optodes covering sensorimotor cortex
• Stimulus presentation: Computer screen with E-Prime or PsychoPy software

II. Participant Preparation

• Recruit healthy adults (N=29 as reference sample size) with minimal motor imagery experience
• Position participant in comfortable chair 60-70 cm from screen
• Apply EEG-fNIRS integrated cap, ensuring proper scalp contact and signal quality
• Verify impedance values <10 kΩ for EEG and adequate signal-to-noise ratio for fNIRS

III. Experimental Procedure

• Trial structure:
  • Cue period (2s): Display black arrow pointing left or right
  • Imagery period (10s): Show fixation cross; participant performs kinesthetic motor imagery (e.g., imagining hand opening/closing at ~1 repetition/second)
  • Rest period (10-12s): Blank screen with fixation cross
• Repeat for 30 trials per condition (left/right hand imagery) in randomized, counterbalanced order

IV. Data Acquisition Parameters

• EEG: Sample at 200 Hz, apply bandpass filter 0.1-100 Hz, include EOG channels for artifact detection
• fNIRS: Record HbO and HbR concentrations at 10 Hz sampling rate using appropriate wavelengths (e.g., 760 nm and 850 nm)
• Synchronization: Use hardware triggers to align EEG and fNIRS data streams with millisecond precision

V. Preprocessing Pipeline

• EEG processing:
  • Downsample to 128 Hz
  • Apply common average reference
  • Bandpass filter (8-25 Hz) to extract μ and low-β bands
  • Remove EOG artifacts using regression or independent component analysis
• fNIRS processing:
  • Convert raw intensity to optical density
  • Filter motion artifacts (e.g., Savitzky-Golay filtering)
  • Convert to hemoglobin concentrations using Modified Beer-Lambert Law
  • Bandpass filter (0.01-0.2 Hz) to remove physiological noise

VI. Feature Extraction

• EEG features: Common Spatial Patterns (CSP) for ERD/ERS quantification
• fNIRS features: Mean and slope values of HbO and HbR during imagery period
• Temporal alignment: Use sliding window analysis (window size=3s, step size=1s) from 5s pre-cue to 20s post-cue

This protocol has demonstrated classification accuracy of approximately 76% using early fusion approaches, significantly outperforming unimodal classification (EEG-only: ~65%, fNIRS-only: ~57%) [15].

Visualization of Experimental Workflows

Figure 1: Comprehensive Workflow for EEG-fNIRS Multimodal Experiment

Clinical Applications and Translational Validation

Diagnostic and Therapeutic Applications

Table 3: Clinical Applications of EEG-fNIRS Fusion Technology

Clinical Domain	Specific Application	Fusion Benefits	Reported Performance
Psychiatry	Schizophrenia diagnosis [111]	Combined neurovascular coupling and electrical activity reveals default mode network abnormalities	Identification of inhibitory PFC→insular connectivity [111]
Epilepsy	Seizure focus localization [5]	Improved spatial precision for EEG spikes with hemodynamic correlation	Enhanced surgical planning accuracy
Neurodegenerative Disorders	Parkinson's disease monitoring [7]	Complementary motor cortex signatures during movement tasks	Successful distinction from healthy controls [7]
Anesthesiology	Depth of anesthesia monitoring [5]	Multimodal confirmation of anesthetic effects on brain function	Reduced incidence of intraoperative awareness
Pediatric Neurology	Infantile spasm evaluation [5]	Non-invasive monitoring suitable for developing brains	Improved seizure characterization
Stroke Rehabilitation	Motor recovery assessment	Tracking both electrical reorganization and hemodynamic adaptation	More comprehensive recovery biomarkers

The translational potential of EEG-fNIRS fusion is particularly promising in psychiatry, where it enables a novel cross-validation framework between clinical phenomenology and neurobiological mechanisms [111]. This approach involves simultaneous administration of psychological assessment tools during neuroimaging acquisition, creating direct "translational bridges" between subjective experience and objective brain measures [112].

For instance, in schizophrenia research, concurrent fMRI and paranoid-depressive scale administration revealed robust activations in default mode network structures (precuneus, posterior cingulate cortex, angular gyrus) during paranoid ideation, alongside aberrant inhibitory connectivity from prefrontal cortex to anterior insula [111]. This methodology can be adapted for EEG-fNIRS studies to establish clinically practical biomarkers for diagnostic assessment and treatment monitoring.

Protocol for Translational Cross-Validation in Psychiatry

Protocol 2: Simultaneous Neuroimaging and Clinical Assessment for Psychiatric Disorders

This protocol adapts the translational cross-validation model proposed by Stoyanov et al. for EEG-fNIRS applications in psychiatric disorders [111] [112].

I. Clinical Assessment Integration

• Selection of validated psychometric scales relevant to target disorder (e.g., paranoid-depressive scale for schizophrenia, HAM-D for depression)
• Synchronized presentation: Scale items displayed visually during neuroimaging acquisition
• Behavioral response recording: Button press or other minimally disruptive response modality
• Real-time monitoring of compliance and engagement

II. Experimental Design Considerations

• Task paradigm: Block design alternating between:
  • Resting state baseline (30s)
  • Neutral control items (30s)
  • Symptom-relevant clinical items (30s)
• Counterbalancing of item presentation to control for order effects
• Inclusion of catch trials to verify attention and task engagement

III. Data Integration and Analysis

• Temporal alignment of hemodynamic/electrophysiological responses with item presentation
• Contrast analysis: [Symptom-relevant items] vs. [Neutral control items]
• Connectivity analysis: Identify network perturbations associated with symptom provocation
• Correlation modeling: Relationship between clinical scale scores and neural response magnitudes

IV. Validation Framework

• Test-retest reliability assessment across multiple sessions
• Specificity testing across diagnostic groups
• Sensitivity to therapeutic interventions (pharmacological, neuromodulation)
• Convergence with other biomarkers (genetic, neuropsychological)

This protocol enables the development of clinically practical biomarkers where validated psychological assessments can eventually serve as proxies for more costly neuroimaging procedures in treatment monitoring and medication selection [112].

Table 4: Key Research Reagent Solutions for EEG-fNIRS Studies

Category	Specific Tool/Resource	Function	Application Notes
Hardware Platforms	NIRScout System (fNIRS)	fNIRS signal acquisition with multiple wavelength capability	Compatible with BrainAMP EEG for synchronized separate systems
	BrainAMP System (EEG)	High-quality EEG acquisition with low noise characteristics
	Integrated EEG-fNIRS Caps	Customizable headgear for simultaneous acquisition	3D-printed or thermoplastic options improve fit and stability
Software Tools	Homer2 / NIRS Toolbox	fNIRS data processing and hemoglobin calculation	Standardized pipeline for optical data preprocessing
	EEGLAB / FieldTrip	EEG signal processing and artifact removal	Comprehensive toolbox for electrophysiological analysis
	Custom Y-shaped Networks	Early fusion deep learning architecture	Implemented in Python (TensorFlow/PyTorch) or MATLAB
Data Resources	Public Dataset A (Shin et al.)	Left/right hand motor imagery benchmark	29 subjects, 30 trials/condition [15]
	Public Dataset B (Shin et al.)	Mental arithmetic vs. relax imagery	Useful for cognitive state classification
Analytical Modules	Common Spatial Patterns (CSP)	EEG feature extraction for motor imagery	Maximizes variance between conditions
	Canonical Correlation Analysis	Multimodal feature fusion	Identifies correlated components across modalities
	Short-Time Fourier Transform	Time-frequency analysis for EEG	Enables spectrogram imaging for deep learning [113]

Methodological Considerations and Validation Standards

Statistical Validation and Cross-Validation Protocols

Rigorous statistical validation is essential given the high-dimensional nature of multimodal neuroimaging data. Recent research highlights critical considerations for cross-validation in neuroimaging-based classification [114]:

• Cross-validation setup significantly impacts statistical significance assessments in model comparison
• Higher numbers of CV folds and repetitions increase likelihood of detecting significant differences, potentially leading to p-hacking
• Recommended practice: Standardize CV procedures (e.g., 5-10 folds with limited repetitions) and report complete methodological details
• Alternative approaches: Nested cross-validation or hold-out test sets provide more reliable performance estimation

Artifact Handling and Data Quality Assurance

Robust artifact removal remains challenging in multimodal studies:

• EEG artifacts: Particularly ocular, muscle, and motion artifacts requiring advanced correction techniques
• fNIRS confounds: Motion artifacts, scalp hemodynamics, and poor optode-scalp coupling
• Emerging solutions: Short-separation fNIRS channels for superficial signal regression [7], parallel EEG-fNIRS artifact detection

Future methodological developments should focus on standardized artifact handling, improved motion correction, and open benchmarking datasets to enhance reproducibility across research sites [7] [5].

The integration of EEG and fNIRS through advanced data fusion techniques represents a transformative approach in neuroscience and clinical medicine. The protocols and methodologies outlined herein provide a framework for rigorous cross-disciplinary validation, enabling researchers to leverage the complementary strengths of electrophysiological and hemodynamic measures. As hardware integration becomes more sophisticated and analytical methods more powerful, this multimodal approach promises to accelerate the translation of basic neuroscience discoveries into clinically actionable tools for diagnosis, therapeutic monitoring, and personalized treatment in neurology and psychiatry.

Conclusion

EEG-fNIRS data fusion represents a transformative approach in neuroimaging that successfully bridges the temporal-spatial resolution divide, enabling more comprehensive brain activity monitoring in naturalistic settings. The integration of these complementary modalities has demonstrated significant improvements in classification accuracy across numerous applications, from brain-computer interfaces to clinical diagnostics, with some studies achieving over 92% classification performance. Future directions should focus on developing more sophisticated unsupervised fusion algorithms, creating larger open-access multimodal datasets, improving hardware portability and integration, and establishing standardized validation frameworks. As these technologies become more accessible and analytical methods more refined, EEG-fNIRS fusion holds tremendous potential to advance personalized medicine, accelerate therapeutic development, and provide deeper insights into brain function across diverse populations and real-world environments. The continued collaboration between engineers, neuroscientists, and clinical researchers will be essential to fully realize the promise of multimodal neuroimaging in understanding and treating neurological and psychiatric disorders.

References

• 1. Concurrent fNIRS and EEG for Brain Function Investigation [https://pmc.ncbi.nlm.nih.gov/articles/PMC9371171/]
• 2. Simultaneous functional near-infrared spectroscopy and ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC5566595/]
• 3. Functional Near-Infrared Spectroscopy and Its Clinical ... [https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2020.00724/full]
• 4. Multimodal fusion of EEG-fNIRS: a mutual information-based ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC7984774/]
• 5. A Cross-Disciplinary Review of the fNIRS-EEG Dual-Modality ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11506513/]
• 6. Multimodal MBC-ATT: cross-modality attentional fusion of ... [https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2025.1660532/full]
• 7. Multimodal fNIRS-EEG sensor fusion: Review of data- ... [https://pubmed.ncbi.nlm.nih.gov/41211102/]
• 8. Preprocessing functional near-infrared spectroscopy ... [https://mne.tools/stable/auto_tutorials/preprocessing/70_fnirs_processing.html]
• 9. Simultaneous multimodal fNIRS-EEG recordings reveal ... [https://www.nature.com/articles/s41598-023-31609-5]
• 10. fNIRS vs EEG: Comparing Non-Invasive Brain Imaging ... [https://imotions.com/blog/learning/research-fundamentals/fnirs-vs-eeg/?srsltid=AfmBOoro2b7yjsOUUx9bjjOuabmJYBkYM6-7tRVbiVd3p60caCT80z1a]
• 11. Temporal vs. spatial resolution and Consumer Neuroscience [https://medium.com/@Pedro_R_Almeida/temporal-vs-spatial-resolution-and-consumer-neuroscience-ca6b360c5890]
• 12. Optimizing spatial specificity and signal quality in fNIRS [https://www.frontiersin.org/journals/neuroergonomics/articles/10.3389/fnrgo.2024.1286586/full]
• 13. Spatial and temporal resolutions of EEG - PubMed Central - NIH [https://pmc.ncbi.nlm.nih.gov/articles/PMC4548479/]
• 14. Enhanced spatiotemporal resolution imaging of neuronal ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC7778454/]
• 15. Early-stage fusion of EEG and fNIRS improves ... [https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.1062889/full]
• 16. Electroencephalography - an overview [https://www.sciencedirect.com/topics/social-sciences/electroencephalography]
• 17. Neurovascular coupling in humans: Physiology ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC4821024/]
• 18. Measurement of Neurovascular Coupling in Neonates - PMC [https://pmc.ncbi.nlm.nih.gov/articles/PMC6387909/]
• 19. A quantitative model for human neurovascular coupling with ... [https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010818]
• 20. EEG–fNIRS signal integration for motor imagery ... [https://www.sciencedirect.com/science/article/pii/S2772528625000299]
• 21. Applications of Functional Near-Infrared Spectroscopy (fNIRS ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC6306799/]
• 22. Functional Magnetic Resonance Imaging (fMRI) - NCBI - NIH [https://www.ncbi.nlm.nih.gov/books/NBK538909/]
• 23. Understanding human brain function in real-world environments [https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3003210]
• 24. Fusion of fNIRS and EEG: a step further in brain research [https://www.bitbrain.com/blog/fusion-fnirs-eeg-brain-activity-research]
• 25. Editorial: Advanced fNIRS applications in neuroscience ... [https://www.frontiersin.org/journals/neurology/articles/10.3389/fneur.2025.1686834/full]
• 26. Evaluating the effects of multimodal EEG-fNIRS ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12425317/]
• 27. Decoding emotions through personalized multi-modal ... [https://www.sciencedirect.com/science/article/pii/S1746809425001430]
• 28. A Decade of Progress: How fNIRS is Transforming Clinical ... [https://www.spectroscopyonline.com/view/a-decade-of-progress-how-fnirs-is-transforming-clinical-brain-imaging]
• 29. Multimodal fNIRS–EEG sensor fusion: Review of data-driven ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12592382/]
• 30. Preprocessing functional Near Infrared Spectroscopy ... [https://nimrobotics.com/blog/2025/fnirs-preprocessing/]
• 31. HEFMI-ICH: a hybrid EEG-fNIRS motor imagery dataset for ... [https://www.nature.com/articles/s41597-025-06100-7]
• 32. A multi-modal dataset of electroencephalography and ... [https://www.nature.com/articles/s41597-025-05286-0]
• 33. Simultaneous EEG-fNIRS study of visual cognitive processing [https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0325017]
• 34. How EEG preprocessing shapes decoding performance [https://www.nature.com/articles/s42003-025-08464-3]
• 35. Early Fusion vs. Late Fusion in Multimodal Data Processing [https://www.geeksforgeeks.org/deep-learning/early-fusion-vs-late-fusion-in-multimodal-data-processing/]
• 36. Multimodal Models and Fusion - A Complete Guide [https://medium.com/@raj.pulapakura/multimodal-models-and-fusion-a-complete-guide-225ca91f6861]
• 37. What fusion strategies work best for combining results from ... [https://milvus.io/ai-quick-reference/what-fusion-strategies-work-best-for-combining-results-from-different-modalities]
• 38. Data Processing in Functional Near-Infrared Spectroscopy ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8151801/]
• 39. Preprocessing functional near-infrared spectroscopy ... [https://mne.tools/1.7/auto_tutorials/preprocessing/70_fnirs_processing.html]
• 40. fNIRS–EEG data preprocessing [https://bio-protocol.org/exchange/minidetail?id=18156217&type=30]
• 41. Feature fusion analysis approach based on synchronous EEG ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11828439/]
• 42. A Topology-Aware Multiscale Feature Fusion Network for ... [https://www.sciencedirect.com/science/article/abs/pii/S0950705125015795]
• 43. Nonlinear neural patterns are revealed in high frequency ... [https://www.sciencedirect.com/science/article/pii/S0361923023001843]
• 44. Improved classification performance of EEG-fNIRS ... [https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2022.973959/full]
• 45. A Decision-Level Fusion Method for COVID-19 Patient Health ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8574072/]
• 46. Research on decision-level fusion method based ... [https://www.sciencedirect.com/science/article/abs/pii/S0952197623012794]
• 47. Hybrid EEG-fNIRS brain-computer interface based on the ... [https://www.sciencedirect.com/science/article/pii/S0208521623000256]
• 48. Multimodal MBC-ATT: cross-modality attentional fusion of ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12504385/]
• 49. Multi-branch GAT-GRU-transformer for explainable EEG- ... [https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2025.1599960/full]
• 50. cross-modality attentional fusion of EEG-fNIRS for cognitive ... [https://pubmed.ncbi.nlm.nih.gov/41070190/]
• 51. Research on drug addiction detection based on AR-TSNET ... [https://biomedical-engineering-online.biomedcentral.com/articles/10.1186/s12938-025-01456-8]
• 52. Multimodal assessment of a BCI system for stroke ... [https://jneuroengrehab.biomedcentral.com/articles/10.1186/s12984-025-01723-8]
• 53. Detecting cognitive motor dissociation by functional near- ... [https://www.frontiersin.org/journals/neurology/articles/10.3389/fneur.2025.1532804/full]
• 54. Unfolding the real-time neural mechanisms in addiction [https://www.sciencedirect.com/science/article/pii/S2772392522000438]
• 55. The ReHand-BCI trial: a randomized controlled ... [https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2025.1579988/full]
• 56. Research on drug addiction detection based on AR-TSNET ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12542443/]
• 57. Research on drug addiction detection based on AR-TSNET ... [https://pubmed.ncbi.nlm.nih.gov/41121050/]
• 58. DeepSyncNet:Deep synchronized fusion network for EEG- ... [https://www.sciencedirect.com/science/article/abs/pii/S1746809425016118]
• 59. Motion artifacts removal and evaluation techniques for ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9576156/]
• 60. Characterizing motion artifacts in functional near-infrared ... [https://www.sciencedirect.com/science/article/pii/S1746809425007670]
• 61. Adaptive filtering of physiological noises in fNIRS data [https://biomedical-engineering-online.biomedcentral.com/articles/10.1186/s12938-018-0613-2]
• 62. Motion Artifacts Correction from Single-Channel EEG and ... [https://www.mdpi.com/1424-8220/22/9/3169]
• 63. Motion Artifact Removal from EEG Signals Using the ... [https://www.sciencedirect.com/science/article/pii/S1746809425013886]
• 64. Signal Processing in fNIRS: A Case for the Removal of ... - PMC [https://pmc.ncbi.nlm.nih.gov/articles/PMC6769087/]
• 65. Physiological Noise Filtering in Functional Near-Infrared ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10295283/]
• 66. Correcting physiological noise in whole-head functional ... [https://www.sciencedirect.com/science/article/pii/S0165027021001977]
• 67. EEG at the Edge: Signals, Selves, and Systems [https://www.scientificarchives.com/article/eeg-at-the-edge-signals-selves-and-systems]
• 68. ninjaCap: a fully customizable and 3D printable headgear ... [https://www.spiedigitallibrary.org/journals/neurophotonics/volume-11/issue-03/036601/ninjaCap--a-fully-customizable-and-3D-printable-headgear-for/10.1117/1.NPh.11.3.036601.full]
• 69. ninjaCap: A fully customizable and 3D printable headgear ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11118375/]
• 70. Design, Manufacturing, and Electroencephalography of the ... [https://www.mdpi.com/2571-5577/8/2/56]
• 71. Automated EEG signal processing: A comprehensive ... [https://www.sciencedirect.com/science/article/abs/pii/S0165027025002055]
• 72. fNIRS vs EEG: Comparing Non-Invasive Brain Imaging ... [https://imotions.com/blog/learning/research-fundamentals/fnirs-vs-eeg/?srsltid=AfmBOorVIoLP1AqASsW_gAzQ8iOqESjoeB4Fwmub0xOUAhEznSJLUtvK]
• 73. Multimodal fNIRS-EEG measurements — Staying in sync [https://www.artinis.com/blogpost-all/2023/multimodal-fnirs-eeg-measurements-staying-in-sync]
• 74. EEG-fNIRS Cookbook [https://www.brainproducts.com/support-resources/eeg-fnirs-cookbook/]
• 75. Anatomical correlations of the international 10–20 sensor ... [https://www.sciencedirect.com/science/article/abs/pii/S105381191400411X]
• 76. Development of an Integrated EEG/fNIRS Brain Function ... [https://www.mdpi.com/1424-8220/21/22/7703/review_report]
• 77. Automated methodology for optimal selection of minimum ... [https://www.nature.com/articles/s41598-022-15252-0]
• 78. a. Head model [https://eeglab.org/tutorials/09_source/Model_Settings.html]
• 79. Towards effective joint learning for EEG-fNIRS integration [https://www.sciencedirect.com/science/article/abs/pii/S0925231225008008]
• 80. Exploring the application and challenges of fNIRS ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10957543/]
• 81. Simultaneous EEG and fNIRS recordings for semantic ... [https://www.nature.com/articles/s41597-025-04967-0]
• 82. A Multimodal EEG-fNIRS Representation-learning Model ... [https://pubmed.ncbi.nlm.nih.gov/41223650/]
• 83. Protocol for Simultaneous EEG and fNIRS Measurement in ... [https://pubmed.ncbi.nlm.nih.gov/40606568/]
• 84. Protocol for Simultaneous EEG and fNIRS Measurement in ... [https://link.springer.com/article/10.1007/s43657-023-00148-4]
• 85. Influence of contrast-reversing frequency on the amplitude ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC6913388/]
• 86. Brain activity characteristics of RGB stimulus: an EEG study [https://www.nature.com/articles/s41598-023-46450-z]
• 87. Motion Artifacts Correction from Single-Channel EEG and ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9102309/]
• 88. Naturalistic approach to investigate the neural correlates of ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11407240/]
• 89. Methods for motion artifact reduction in online brain ... [https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2023.1251690/full]
• 90. Learning based motion artifacts processing in fNIRS: a mini ... [https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2023.1280590/full]
• 91. Naturalistic fNIRS assessment reveals decline in executive ... [https://www.nature.com/articles/s41598-025-20844-7]
• 92. Improved classification performance of EEG-fNIRS multimodal ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9388144/]
• 93. Feature Extraction and Classification Methods for Hybrid ... [https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2018.00246/full]
• 94. Classification: Accuracy, recall, precision, and related metrics [https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall]
• 95. Understanding F1 Score, Accuracy, ROC-AUC & PR- ... [https://www.deepchecks.com/f1-score-accuracy-roc-auc-and-pr-auc-metrics-for-models/]
• 96. F1 Score vs ROC AUC vs Accuracy vs PR AUC [https://neptune.ai/blog/f1-score-accuracy-roc-auc-pr-auc]
• 97. A quantitative comparison of NIRS and fMRI across multiple ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC3021967/]
• 98. Neurophysiological data augmentation for EEG-fNIRS ... [https://www.sciencedirect.com/science/article/abs/pii/S0169260725000112]
• 99. EFRM: A Multimodal EEG–fNIRS Representation-learning ... [https://www.sciencedirect.com/science/article/abs/pii/S0010482525016464]
• 100. Benchmarking framework for machine learning ... [https://www.frontiersin.org/journals/neuroergonomics/articles/10.3389/fnrgo.2023.994969/full]
• 101. REFED2025/REFED-dataset [https://huggingface.co/datasets/REFED2025/REFED-dataset]
• 102. An EEG-fNIRS neurovascular coupling analysis method to investigate cognitive-motor interference - ScienceDirect [https://www.sciencedirect.com/science/article/abs/pii/S001048252300433X]
• 103. Automatic depression diagnosis through hybrid EEG and ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC10483285/]
• 104. Validation of fNIRS measurement of executive demand ... [https://www.sciencedirect.com/science/article/pii/S2213158224000767]
• 105. Machine learning in attention-deficit/hyperactivity disorder [https://www.nature.com/articles/s41398-023-02536-w]
• 106. Application and future directions of brain-computer ... [https://www.sciencedirect.com/science/article/pii/S2589238X2500083X]
• 107. The top BCI stories of 2025 so far [https://www.massdevice.com/top-bci-stories-2025-so-far/]
• 108. Decoding motor execution and motor imagery from EEG ... [https://www.sciencedirect.com/science/article/pii/S266709922500012X]
• 109. Motor imagery decoding network with multisubject dynamic ... [https://braininformatics.springeropen.com/articles/10.1186/s40708-025-00267-w]
• 110. TFANet: a temporal fusion attention neural network for ... [https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2025.1635588/full]
• 111. Implications from translational cross-validation of clinical ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8134869/]
• 112. AS09-02 - Translational cross-validation among ... [https://www.sciencedirect.com/science/article/pii/S0924933812739767]
• 113. Simultaneous EEG-fNIRS Data Classification Through ... [https://pubmed.ncbi.nlm.nih.gov/39247844/]
• 114. Statistical variability in comparing accuracy of ... [https://www.nature.com/articles/s41598-025-12026-2]