[ICML 2025]MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from

①Limitations of alignment in cross-subject model: scarce data in shared space construction, and subject respond differently in watching the same image

②Conception of MindAligner:

2.3. Related Work

2.3.1. fMRI-Based Brain Decoding

①Lists traditional methods and point out their work might be inefficient in cross subject reconstruction

2.3.2. Cross-Subject Functional Alignment

①Align limits inter subject specificity

2.4. Preliminary

①Only use one hour of data to pretrain, then test model in shared test set

2.5. MindAligner

2.5.1. Overview

①The overall framework of MindAligner:

2.5.2. Brain Transfer Matrix

①Given the the fMRI signal $\mathcal{F}_{N}$ for a subject $S_N$ , the brain transfer matrix (BTM) maps it to:

$\hat{\mathcal{F}_K}=\mathcal{M}\times\mathcal{F}_N$

②The $\mathcal{M}$ can be decomposed to two low-rank matrices:

$\mathcal{M}=\mathcal{A}\times\mathcal{B}$

where $\mathcal{A}\in\mathbb{R}^{n\times h}$ and $\mathcal{B}\in\mathbb{R}^{h\times k}$ , $n$ and $k$ denotes the fMRI voxel dimensions of unkown and kown subjects, $h$ is the hidden dimension

2.5.3. Brain Functional Alignment Module

①Generate the stimuli embedding $z_K$ of unkown subject by stimuli differential condition to align kown embedding $\mathcal{F}_N$ :

$z_{N}=\mathcal{A}\times\mathcal{F}_{N}$

$\begin{aligned} & E_{\mathrm{diff}}=\mathcal{E}_{image}(\mathcal{I}_{N})-\mathcal{E}_{image}(\mathcal{I}_{K}), \\ & z_{\mathrm{diff}}=E_\mathrm{diff}\times\mathcal{M}_\mathrm{diff}, \\ & \boldsymbol{z}_{K}=\mathcal{M}_{C}(z_{N},z_{\mathrm{diff}}), \end{aligned}$

where $\mathcal{E}_{image}$ is pretrained CLIP, as the image encoder. $\mathcal{M}_{C}$ is the cross-stimulus neural mapper, 它使用 $\mathcal{M}_{\mathrm{diff}}\in\mathbb{R}^{a\times2h}$ 将条件 $z_{\mathrm{diff}}$ 分解为缩放和移位参数??

②Further align by:

$\hat{\mathcal{F}}_{K}=z_{K}\times\mathcal{B}.$

③Reconstruction loss:

$\mathcal{L}_{rec}=||\mathcal{F}_{K}-\mathcal{F}_{K}||_{2}^{2}$

and distribution loss:

$\mathcal{L}_{KL}=\mathcal{KL}(\mathcal{F}_{K},\mathcal{F}_{K})$

④Loss between fMRI embedding pairs and stimuli pairs:

$\mathcal{L}_{latent}=\|(\mathcal{R}(\mathcal{E}_{f}(\boldsymbol{z}_{N}),\mathcal{E}_{f}(\boldsymbol{z}_{K}))-\mathcal{R}(\boldsymbol{E}_{N},\boldsymbol{E}_{K})\|_{2}^{2}$

where $E_N$ and $E_K$ are the image embeddings from CLIP, $\mathcal{R}(\cdot)$ denotes dissimilarity function

⑤The final loss:

$\mathcal{L}_{\mathrm{Align}}=\mathcal{L}_{Dec}+\alpha_{rec}\mathcal{L}_{rec}+\alpha_{KL}\mathcal{L}_{KL}+\alpha_{la}\mathcal{L}_{latent},$

where $\mathcal{L}_{Dec}$ denotes the decoding loss in the baseline method

2.5.4. Inference

①BTM only

2.6. Experiments

2.6.1. Implementation Details

①BTM: consist of 2 linear layers with hidden dim of 4096

②Dimension of $\mathcal{M}_\mathrm{diff}$ : 768

③Input and output dimension of functional embedder: 4096

④Loss coefficients: $\alpha_{rec}=1,\alpha_{la}=\alpha_{KL}=0.001,\alpha_{1}=0.033,\alpha_{2}=0.016$

⑤Learning rate: 1e-5

⑥Batch size: 16

2.6.2. Dataset

①Dataset: NSD

2.6.3. Metrics

①Lists metrics for performance and functional alignment measurement

2.6.4. fMRI-based Visual Decoding

①Visualization of reconstructed image:

②Quantitative performance:

③Loss ablation:

④Ablation of alignment:

⑤Parameter comparison:

2.6.5. Brain Functional Alignment Analysis

①Visualization transfer quantity in brain:

they define that early visualization region presents lower inter-subject variability, and higher visual regions (including OPA, FFA, PPA, and EBA) show larger variability

②Performance of different alignment: