Skip to content

Models for predicting fMRI data associated with the Algonauts 2021 challenge (http://algonauts.csail.mit.edu).

License

Notifications You must be signed in to change notification settings

francesco-innocenti/algonauts-2021

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

66 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Algonauts 2021 Models

This is a repo evaluating models for predicting fMRI data associated with the Algonauts Project 2021 Challenge (Cichy et al., 2021; Cichy, Roig & Oliva, 2019). In brief, the challenge is to predict the fMRI responses of 10 participants to over 1000 short natural videos.

In this mini-project - which was conducted as part of Neuromatch Academy 2021 - we were interested in testing a biologically inspired model. We compared PredNet (Lotter, Kreiman & Cox, 2016, 2020), a network inspired by the influential neuroscience theory of predictive coding (Millidge, Seth & Buckley, 2021), to VGG19 (Simonyan & Zisserman, 2015), currently one of the most predictive models of the ventral visual stream (Nonaka et al., 2021, Schrimpf et al., 2020).

We used a PredNet pretrained on a self-driving car dataset (see Lotter, Kreiman & Cox, 2016 for details). Here are some example predictions of this model on the Algonauts videos.

Alt Text Alt Text Alt Text Alt Text Alt Text Alt Text

Alt Text Alt Text Alt Text Alt Text Alt Text Alt Text

We built two simple encoding models (Naselaris et al., 2011; van Gerven, 2017), extracting the layer activations of each network to 900 training videos, reducing their dimensionality with principal component analysis, and linearly regressing the components onto the fMRI responses of 9 visual regions in all subjects. We evaluated the fitted models on a held-out validation set of 100 videos. Here are the results.

Alt Text

Interestingly, we found that VGG19 - which has about 3 orders of magnitude more parameters than PredNet - needed many more layers to match the performance of PredNet, suggesting that the latter is much more efficient.

Code organisation

The code for the two models, PredNet and VGG19, is stored in different subdirectories (prednet and vgg19) because they have different dependencies. Functions used for computation with both models are stored in a custom-made mini-package called algonauts.

Releases

No releases published

Packages

No packages published

Languages