fMRI Encoding of Visual Cortex with Contrastive Learning

• Alex Mulrooney, Electrical Engineering, University of Delaware

• Austin Brockmeier, Electrical and Computer Engineering and Computer and Information Sciences (Joint), University of Delaware

Recent advances in computing power and neuroimaging data collection allow for new avenues of research into building computational models of the brain. Previous efforts to predict brain activity caused by visual stimuli in the visual cortex have typically relied on fixed nonlinear feature extraction from the images. Using the rich and extensive Natural Scenes Dataset, we introduce a method of fine-tuning a pretrained convolutional neural network (CNN) using a contrastive loss function such to more closely align the CNN’s image representations to the brain’s representations. We train models with this approach that are specific to subjects and regions of interest (ROI). We show that this method achieves significantly higher encoding accuracy as compared to both using the features from the untuned CNN as well as a CNN that is fine-tuned with a simple regression approach. The contrastive learning-based approach does notably better in ROIs higher in the visual processing stream. Furthermore, we evaluate how well the subject-specific representations learned by the models transfer to predicting activations in the same ROI in other subjects. We find that in general, the representations are highly transferable and minimal losses in encoding accuracy are incurred from such a cross-subject approach. These results suggest that CNNs can be made more “brain-like” by fine tuning them on human neural data so that the image representations in the artificial network are more aligned with the representations in the biological human network.