This month: Microsoft, INRI and IIIT tackle the Brain-computer 💻 interface and Visio-linguistic Transformers to solve the brain encoding problem 🧮 and the very first public graph neural network framework 🕸 ️ to explore brain structural and functional networks.

Support my writing and projects by joining Medium through my referral link:
Why should you care about Neuroscience?
Neuroscience is the root of nowadays artificial intelligence 🧠🤖. Reading and being aware of the evolution and new insights in neuroscience not only will allow you to be a better "Artificial Intelligence" guy 😎 , but also a finer neural network architectures creator 👩 💻 !
In this post, I am officially relaunching my "Stay Updated With Neuroscience"articles with a new approach. In this series, I will cover 3 main papers, under review on arxiv.org which deal with Machine Learning and neuroscience. In particular, I will cover the following aspects:
- can ML research help neuroscience in getting a deeper understanding of the brain’s dynamics and activities?
- how neuroscience can help enhance ML with new biologically inspired models?
- how ML and models can give us new clinical neuroscience, with new imaging and signal techniques?
This month I’ll share with you 2 papers that caught my attention in the ML/neuroscience world. The former comes from a collaboration between Microsoft India and INRI and IIIT and it’s a remarkable job to investigate powerful deep learning techniques to solve the brain encoding problem. Brain encoding is a core problem to be solved to allow brain-computer interface systems. Authors have investigated, for the very first time in the field, the application of Visio-linguistic Transformers to decouple fMRI signals and brain regions activations. The second paper is a great collaboration that goes from Emory University and the University of Pittsburgh. Authors tested and released to the public BrainGB, a big brain graph neural network system, which allows researchers to look for predictions on clinical outcomes as well as brain biomarkers. This is a pivotal work, which paves the road and covers the gap between neuroscience and deep learning to get more and more insights into brain structures from a graph neural network perspective. Simply amazing.
Visio-Linguistic Brain Encoding
Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, Bapi Raju Surampudi, Paper
Can ML research help neuroscience in getting a deeper understanding of the brain’s dynamics and activities?
This paper comes from a collaboration between INRIA, IIIT, and Microsoft India, addressing one of the hottest topics of research of the moment: brain-computer interface and brain encoding. To enable a fully effective brain-computer interface system, we do need to understand how the human brain encodes stimuli. Given a visual or a language stimulus the brain performs a "mysterious" encoding to translate the signal into brain activity. In this work, the authors have investigated whether image-based and multi-model Transformers can encode fMRI signals on the whole brain and how accurately the encoding is done.

Until now, brain encoding research has been limited to Convolutional Neural Networks (CNNs) and two single brain regions, the visual cortex V4 and the pre-frontal IT. CNNs showed to encode semantics from visual stimuli effectively, however, there is no understanding of how the brain as a whole can encode. Moreover, investigated CNNs were carefully tuned with a lot of manual processing before receiving meaningful results.
To investigate these problems, the authors have employed 2 neuroscience datasets: BOLD5000 and Pereira. The former is a dataset of fMRI scans collected from subjects viewing 5254 natural images and it points out 5 visual areas: the early visual area (EarlyVis), object-related areas such as lateral occipital complex (LOC), occipital place area (OPA), parahippocampal place area (PPA) and retrosplenial complex (RSC). The latter dataset reports fMRI scans from subjects visualizing concept words along with a picture. The main focus of this dataset is on Default Mode Network (DMN), Language Network, Task Positive Network, Visual Network
Models (pertained CNNs – VGNet19, ResNet50, INceptionV2ResNet, EfficientNetB5, pretrained text Transformers – RoBERTa, image Transformers – ViT, DEiT, BEiT, late-fusion models – VGGNEt19+RoBERTa, ReNet50 + RoBERTa, InceptionV2ResNet+RoBERTa, EffficientNetB5+RoBERTa, multi-model Transformers – CLIP, LXMERT, VisualBERT.) were trained on the fMRI scans, with a ridge regression loss. The main goal for each model is to encode and predict fMRI voxel values for each brain region given a stimulus.
Results clearly show that VisualBERT outperforms all the other models. For BOLD5000 VisualBERT outperforms all the models for both 2V2 accuracy and Pearson correlation (fig.2). in particular VisualBERT shows a high Pearson R for OPA and LOC areas, in line with the visual processing hierarchy – there is a joint encoding (visual and language information). On the Pereira, dataset VisualBERT displays a high correlation with Language regions, DMN and TP.

We can conclude that multi-model Visio-linguistic Transformers outperform current ML methods employed to decypher the brain encoding problem. These new insights are paving a new way for fMRI analyses and the road to a brain-computer interface
BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networks
Hejie Cui, Wei Dai, Yanqiao Zhu, Xuan Kan, Antonio Aodong Chen Gu, Joshua Luemire, Liang Zhan, Lifang He, Ying Gut, Carl Yang, Paper

How ML and models can give us new clinical Neuroscience, with new imaging and signal techniques?
This is a pivotal paper, as it fills the gap between brain connectome and graph neural networks (GNN). In particular, the authors present BrainGB, a unified, modular, scalable, and reproducible framework for brain network analysis with GNNs.
Understanding structures, functions, and brain dynamics is intriguing research in neuroscience. From here we could get the insights for various goals, like mental disorder therapies or achieving general artificial intelligence. Moreover, the interaction between various brain regions is a driving factor for neural development or disorders. If we could translate the brain into a graph system, with nodes and edges, we could start getting a more insightful understanding of some neural mechanisms. Over the years different experimental techniques have been used to map the brain to a graph. Data from Magnetic-Resonance Imaging (MRI) or Electroencephalography (EEG) or Positron Emission Tomography (PET) or Diffusion Tensor Imaging (DTI) have been used and studied to draw a brain network, which could describe correlations across brain regions signals, through gray matter regions connectivity. Different attempts have been made to create a full network using Machine Learning. Several studies have tried to predict brain disease by learning the brain network graph structure. However, many proposed experimental protocols are limited to local datasets, or, due to ethical reasons, are not publicly available. Often, details of imaging processing are not disclosed, making it harder to evaluate results and reproduce results. For these reasons, authors have created BrainGB, a public benchmark platform, to evaluate deep graph models for brain network analysis.
Fig.1 reports the final BrainGB structure, which is made of these 3 points:
- The framework is unified, modular, scalable, and reproducible. The design ensures a fair evaluation, publicly available datasets, hyperparameters, settings and baselines
- Both functional and structural brain networks are used to cover the gap between neuroimaging and deep learning
- GNN approach has been subdivided into 4 steps: 1) node features, 2) message passing 3) attention mechanism 4) pooling strategies
The project main code is available at: https://github.com/HennyJie/BrainGB and tutorials can be found on https://brainnet.us .


To build up the brain network, functional and structural information has been extracted from fMRI and dMRI scans. Figs. 4a and 4b report the preprocessing protocol along with input software employed to extract all the relevant information.
Once raw information different GNN modular designs have been applied. The first approach is a pure graph neural network, with node feature construction. This first pass feature records the high-dimensional identity for each brain network region of interest (ROI). The second approach employed message passing, where the node representation is updated iteratively by aggregating neighbor features through local connections. The third pass is based on the attention mechanism, where the model updates the brain region
representation in a data-driven way. The final step is using pooling strategies. To test out predictions from BrainGB authors have run a series of validations against four datasets built up from fMRI and dMRI scans (HIV, PNC, PPMI, and ABCD datasets). For each dataset, the accuracy, F1, and AUC score were recorded and evaluated for each of the 4 graph strategies. There is a little surprise but the attention mechanism based on a good node concatenation returns the best results, compared to the other strategies (tab.1, results for HIV only)
The authors here provide the very first. unified, module, scalable and reproducible framework for using brain network analysis merged with graph neural network. From here researchers can have access to reproducible datasets, baselines, and tutorials as well as asking a model to run predictions on clinical outcomes that can be improved day by day.
If you have questions or curiosities, just send an email to: [email protected] or follow me on Medium 🙂