Neurons is a novel framework that emulates the human visual cortex to achieve high-fidelity and interpretable fMRI-to-video reconstruction. Our biologically inspired approach significantly advances the state-of-the-art in brain decoding and visual reconstruction.

🟡 2025/10    Released model weights, training logs, testing logs, and generated images and videos — all available at Hugging Face! ⚠️ Note 1: Cloning the entire EXP folder requires over 60 GB of storage. To download selectively, use snapshot_download with the allow_patterns parameter (e.g., allow_patterns=["EXP/exp_neurons/subj_1/*"] to download only subject 1). ⚠️ Note 2: Due to a server issue, the original weights were lost. We re-cloned the repository and re-ran the experiments. While specific numerical results may vary slightly, the overall performance remains consistent with the paper, which also verifies the reproducibility of this work.

🟡 2025/06    Neurons is accepted by ICCV-2025!

🟡 2025/04    Code released!

🟡 2025/03    Project launched with paper available on arXiv!

We recommend using separate environments for training and testing:

# Training environment
conda create -n neurons_train python==3.10
conda activate neurons_train
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

# Testing environment (to avoid package conflicts)
conda create -n neurons_test --clone neurons_train
conda activate neurons_test
pip install diffusers==0.11.1

1. Download the pre-processed dataset:

python download_dataset.py
tar -xzvf ./cc2017_dataset/masks/mask_cls_train_qwen_video.tar.gz -C ./cc2017_dataset/masks/
tar -xzvf ./cc2017_dataset/masks/mask_cls_test_qwen_video.tar.gz -C ./cc2017_dataset/masks/

1. Run task construction scripts:

# Rule-based Key Object Discovery
python tasks_construction/find_key_obj.py

# Generate CLIP embeddings
python -m tasks_construction.gen_GT_clip_embeds

mkdir pretrained_weights
cd pretrained_weights
wget -O unclip6_epoch0_step110000.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/unclip6_epoch0_step110000.ckpt\?download\=true
wget -O last.pth -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/train_logs/final_subj01_pretrained_40sess_24bs/last.pth\?download\=true
wget -O convnext_xlarge_alpha0.75_fullckpt.ckpt -c https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/convnext_xlarge_alpha0.75_fullckpt.pth\?download\=true
wget -O sd_image_var_autoenc.pth https://huggingface.co/datasets/pscotti/mindeyev2/resolve/main/sd_image_var_autoenc.pth\?download\=true
cd ..

This codebase allows train, test, and evaluate using one single bash file.

bash train_neurons.sh 0 neurons 123456 enhance 1

Parameters:

$1: use which gpu to train

$2: train file postfix, e.g, train_neurons

$3: run which stage: 123456 for the whole process, 3456 for test & eval only

• 1: train brain model
• 2: train decoupler
• 3: recon decoupled outputs, prepare for video reconstruction
• 4: (Optional) caption the keyframes with BLIP-2 instead of using the outputs of GPT-2 in Neurons
• 5: video reconstruction
• 6: evaluation with all metrics

$4: inference mode: ['enhance', 'motion']

$5: train which subject: [0,1,2]

Note that for convenience of debugging, use_wandb is set to False be default.

If you would like to use wandb, first run wandb login and set the use_wandb to True in train_neurons.py.

If you find this project useful, please consider citing:

@inproceedings{wang2025neurons,
  title={Neurons: Emulating the human visual cortex improves fidelity and interpretability in fmri-to-video reconstruction},
  author={Wang, Haonan and Zhang, Qixiang and Wang, Lehan and Huang, Xuanqi and Li, Xiaomeng},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={18367--18376},
  year={2025}
}