The world’s leading publication for data science, AI, and ML professionals.

ML & Nuroscience: April 2022 must-reads

This month: Microsoft, INRI and IIIT tackle the brain-computer 💻 interface and Visio-linguistic Transformers to solve the brain encoding…

This month: Microsoft, INRI and IIIT tackle the Brain-computer 💻 interface and Visio-linguistic Transformers to solve the brain encoding problem 🧮 and the very first public graph neural network framework 🕸 ️ to explore brain structural and functional networks.

Image by Mo on Unsplash
Image by Mo on Unsplash

Support my writing and projects by joining Medium through my referral link:

Join Medium with my referral link – Stefano Bosisio

Why should you care about Neuroscience?

Neuroscience is the root of nowadays artificial intelligence 🧠🤖. Reading and being aware of the evolution and new insights in neuroscience not only will allow you to be a better "Artificial Intelligence" guy 😎 , but also a finer neural network architectures creator 👩 ‍💻 !

In this post, I am officially relaunching my "Stay Updated With Neuroscience"articles with a new approach. In this series, I will cover 3 main papers, under review on arxiv.org which deal with Machine Learning and neuroscience. In particular, I will cover the following aspects:

  • can ML research help neuroscience in getting a deeper understanding of the brain’s dynamics and activities?
  • how neuroscience can help enhance ML with new biologically inspired models?
  • how ML and models can give us new clinical neuroscience, with new imaging and signal techniques?

This month I’ll share with you 2 papers that caught my attention in the ML/neuroscience world. The former comes from a collaboration between Microsoft India and INRI and IIIT and it’s a remarkable job to investigate powerful deep learning techniques to solve the brain encoding problem. Brain encoding is a core problem to be solved to allow brain-computer interface systems. Authors have investigated, for the very first time in the field, the application of Visio-linguistic Transformers to decouple fMRI signals and brain regions activations. The second paper is a great collaboration that goes from Emory University and the University of Pittsburgh. Authors tested and released to the public BrainGB, a big brain graph neural network system, which allows researchers to look for predictions on clinical outcomes as well as brain biomarkers. This is a pivotal work, which paves the road and covers the gap between neuroscience and deep learning to get more and more insights into brain structures from a graph neural network perspective. Simply amazing.

Visio-Linguistic Brain Encoding

Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, Bapi Raju Surampudi, Paper

Can ML research help neuroscience in getting a deeper understanding of the brain’s dynamics and activities?

This paper comes from a collaboration between INRIA, IIIT, and Microsoft India, addressing one of the hottest topics of research of the moment: brain-computer interface and brain encoding. To enable a fully effective brain-computer interface system, we do need to understand how the human brain encodes stimuli. Given a visual or a language stimulus the brain performs a "mysterious" encoding to translate the signal into brain activity. In this work, the authors have investigated whether image-based and multi-model Transformers can encode fMRI signals on the whole brain and how accurately the encoding is done.

Fig.1: Proposed brain encoding methodology. Visual and text stimuli are the input for multi-model Transformers. From here the model predicts the fMRI aactivations for different brain regions. Brain encoding results are then evaluated with 2V2 accuracy and Pearson correlation coefficient (R).
Fig.1: Proposed brain encoding methodology. Visual and text stimuli are the input for multi-model Transformers. From here the model predicts the fMRI aactivations for different brain regions. Brain encoding results are then evaluated with 2V2 accuracy and Pearson correlation coefficient (R).

Until now, brain encoding research has been limited to Convolutional Neural Networks (CNNs) and two single brain regions, the visual cortex V4 and the pre-frontal IT. CNNs showed to encode semantics from visual stimuli effectively, however, there is no understanding of how the brain as a whole can encode. Moreover, investigated CNNs were carefully tuned with a lot of manual processing before receiving meaningful results.

To investigate these problems, the authors have employed 2 neuroscience datasets: BOLD5000 and Pereira. The former is a dataset of fMRI scans collected from subjects viewing 5254 natural images and it points out 5 visual areas: the early visual area (EarlyVis), object-related areas such as lateral occipital complex (LOC), occipital place area (OPA), parahippocampal place area (PPA) and retrosplenial complex (RSC). The latter dataset reports fMRI scans from subjects visualizing concept words along with a picture. The main focus of this dataset is on Default Mode Network (DMN), Language Network, Task Positive Network, Visual Network

Models (pertained CNNs – VGNet19, ResNet50, INceptionV2ResNet, EfficientNetB5, pretrained text Transformers – RoBERTa, image Transformers – ViT, DEiT, BEiT, late-fusion models – VGGNEt19+RoBERTa, ReNet50 + RoBERTa, InceptionV2ResNet+RoBERTa, EffficientNetB5+RoBERTa, multi-model Transformers – CLIP, LXMERT, VisualBERT.) were trained on the fMRI scans, with a ridge regression loss. The main goal for each model is to encode and predict fMRI voxel values for each brain region given a stimulus.

Results clearly show that VisualBERT outperforms all the other models. For BOLD5000 VisualBERT outperforms all the models for both 2V2 accuracy and Pearson correlation (fig.2). in particular VisualBERT shows a high Pearson R for OPA and LOC areas, in line with the visual processing hierarchy – there is a joint encoding (visual and language information). On the Pereira, dataset VisualBERT displays a high correlation with Language regions, DMN and TP.

Fig.2: MAE between real and predicted fMRI voxels: a) V2 and V3 regions for VisualBERT on the BOLD5000 subject 1. b) VisualBERT on the Pereira dataset subject 2.
Fig.2: MAE between real and predicted fMRI voxels: a) V2 and V3 regions for VisualBERT on the BOLD5000 subject 1. b) VisualBERT on the Pereira dataset subject 2.

We can conclude that multi-model Visio-linguistic Transformers outperform current ML methods employed to decypher the brain encoding problem. These new insights are paving a new way for fMRI analyses and the road to a brain-computer interface

BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networks

Hejie Cui, Wei Dai, Yanqiao Zhu, Xuan Kan, Antonio Aodong Chen Gu, Joshua Luemire, Liang Zhan, Lifang He, Ying Gut, Carl Yang, Paper

Fig.3: the BrainGB framework. A graph neural network receives as input fMRI scans, sMRI, dMRI and creates a network structure. From here users will be able to obtain predictions for clinical outcomes as well as biomarkers.
Fig.3: the BrainGB framework. A graph neural network receives as input fMRI scans, sMRI, dMRI and creates a network structure. From here users will be able to obtain predictions for clinical outcomes as well as biomarkers.

How ML and models can give us new clinical Neuroscience, with new imaging and signal techniques?

This is a pivotal paper, as it fills the gap between brain connectome and graph neural networks (GNN). In particular, the authors present BrainGB, a unified, modular, scalable, and reproducible framework for brain network analysis with GNNs.

Understanding structures, functions, and brain dynamics is intriguing research in neuroscience. From here we could get the insights for various goals, like mental disorder therapies or achieving general artificial intelligence. Moreover, the interaction between various brain regions is a driving factor for neural development or disorders. If we could translate the brain into a graph system, with nodes and edges, we could start getting a more insightful understanding of some neural mechanisms. Over the years different experimental techniques have been used to map the brain to a graph. Data from Magnetic-Resonance Imaging (MRI) or Electroencephalography (EEG) or Positron Emission Tomography (PET) or Diffusion Tensor Imaging (DTI) have been used and studied to draw a brain network, which could describe correlations across brain regions signals, through gray matter regions connectivity. Different attempts have been made to create a full network using Machine Learning. Several studies have tried to predict brain disease by learning the brain network graph structure. However, many proposed experimental protocols are limited to local datasets, or, due to ethical reasons, are not publicly available. Often, details of imaging processing are not disclosed, making it harder to evaluate results and reproduce results. For these reasons, authors have created BrainGB, a public benchmark platform, to evaluate deep graph models for brain network analysis.

Fig.1 reports the final BrainGB structure, which is made of these 3 points:

  • The framework is unified, modular, scalable, and reproducible. The design ensures a fair evaluation, publicly available datasets, hyperparameters, settings and baselines
  • Both functional and structural brain networks are used to cover the gap between neuroimaging and deep learning
  • GNN approach has been subdivided into 4 steps: 1) node features, 2) message passing 3) attention mechanism 4) pooling strategies

The project main code is available at: https://github.com/HennyJie/BrainGB and tutorials can be found on https://brainnet.us .

Fig. 4a: fMRI data preprocessing to build up the brain network, along with a series of softwares employed to extract information
Fig. 4a: fMRI data preprocessing to build up the brain network, along with a series of softwares employed to extract information
Fig. 4b: dMRI data processing to extract brain structural information along with software employed to extract information.
Fig. 4b: dMRI data processing to extract brain structural information along with software employed to extract information.

To build up the brain network, functional and structural information has been extracted from fMRI and dMRI scans. Figs. 4a and 4b report the preprocessing protocol along with input software employed to extract all the relevant information.

Once raw information different GNN modular designs have been applied. The first approach is a pure graph neural network, with node feature construction. This first pass feature records the high-dimensional identity for each brain network region of interest (ROI). The second approach employed message passing, where the node representation is updated iteratively by aggregating neighbor features through local connections. The third pass is based on the attention mechanism, where the model updates the brain region

representation in a data-driven way. The final step is using pooling strategies. To test out predictions from BrainGB authors have run a series of validations against four datasets built up from fMRI and dMRI scans (HIV, PNC, PPMI, and ABCD datasets). For each dataset, the accuracy, F1, and AUC score were recorded and evaluated for each of the 4 graph strategies. There is a little surprise but the attention mechanism based on a good node concatenation returns the best results, compared to the other strategies (tab.1, results for HIV only)

The authors here provide the very first. unified, module, scalable and reproducible framework for using brain network analysis merged with graph neural network. From here researchers can have access to reproducible datasets, baselines, and tutorials as well as asking a model to run predictions on clinical outcomes that can be improved day by day.


If you have questions or curiosities, just send an email to: [email protected] or follow me on Medium 🙂


Related Articles