license: other
license_name: fair-noncommercial-research-license
license_link: https://huggingface.co/facebook/dynadiff/blob/main/LICENSE
extra_gated_prompt: >-
Please be sure to provide your full legal name, date of birth, and full
organization name with all corporate identifiers. Avoid the use of acronyms
and special characters. Failure to follow these instructions may prevent you
from accessing this model and others on Hugging Face. You will not have the
ability to edit this form after submission, so please ensure all information
is accurate.
extra_gated_fields:
First Name: text
Last Name: text
Date of birth: date_picker
Country: country
Affiliation: text
I accept the terms and conditions: checkbox
geo: ip_location
language:
- en
tags:
- neural-decoding
- fMRI
- brain-to-image
Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI
This repository contains the model weights for our paper: "Single-stage Decoding of Images from Continuously Evolving fMRI".
Abstract
Brain-to-image decoding has been recently propelled by the progress in generative AI models and the availability of large ultra-high field functional Magnetic Resonance Imaging (fMRI). However, current approaches depend on complicated multi-stage pipelines and preprocessing steps that typically collapse the temporal dimension of brain recordings, thereby limiting time-resolved brain decoders. Here, we introduce Dynadiff (Dynamic Neural Activity Diffusion for Image Reconstruction), a new single-stage diffusion model designed for reconstructing images from dynamically evolving fMRI recordings. Our approach offers three main contributions. First, Dynadiff simplifies training as compared to existing approaches. Second, our model outperforms state-of-the-art models on time-resolved fMRI signals, especially on high-level semantic image reconstruction metrics, while remaining competitive on preprocessed fMRI data that collapse time. Third, this approach allows a precise characterization of the evolution of image representations in brain activity. Overall, this work lays the foundation for time-resolved brain-to-image decoding.
To run the model, see the readme here: https://github.com/facebookresearch/dynadiff