Sentiment Analysis and Emotion Recognition in Italian (using BERT)

Thoughts and Theory

Introducing FEEL-IT: a data set and a package for sentiment analysis and emotion recognition in Italian.

Recognizing emotions in text is fundamental to get a better sense of how people are talking about something. People can talk about a new event, but positive/negative labels might not be enough. There is a big difference between being angered by something and scared by something. This difference is why it is vital to consider sentiment and emotion in text.

There is a lot of research on sentiment analysis and emotion recognition…for English. A quick search on Google will bring you to different possible algorithms that can take care of sentiment/emotion prediction for you. However, some languages lack data, and one of these languages is Italian (but there is some work, for example, Sprugnoli, 2020).

So we (Debora, Dirk, and Yours Truly) tried to provide a solution to this problem. We created a new data set for Italian sentiment and emotion prediction and fine-tuned a Bert model.

In this blog post, we describe our adventure in creating this new data set, explaining a few things we did and our results. What I am describing here is also available in a peer-reviewed research paper:

Federico Bianchi, Debora Nozza and Dirk Hovy (2021). FEEL-IT: Emotion and Sentiment Classification for the Italian Language. 11th Workshop on Computational Approaches to Subjectivity, Sentiment, and Social Media Analysis.

Since the ready availability of these methods is essential for the community, you can find online:

• our python package
• our HuggingFace sentiment model
• our HuggingFace emotion model

FEEL-IT: Emotion and Sentiment Classification for the Italian Language

Building a New Data Set for Sentiment and Emotion Recognition

One of the issues that we need to address when creating a new data set is that it needs to be representative of the domain. One cannot collect two days of tweets with three keywords. The collected tweets would be too domain-dependent, making the trained models not general enough to be applied to different domains.

We took a different approach instead: we decided to collect 1000 tweets each day related to the trending topics on Twitter. In this way, we are getting the most relevant tweets of each day, and since every day there is something new to talk about, we collect a lot of different stuff. This approach allowed us to get a data set that contains tweets from a wide range of topics: from football matches to politics.

Then, we annotated the tweets with emotions, excluding those tweets that had no recognizable emotion. Eventually, we got to 2037 annotated tweets: 912 anger, 103 fear, 728 joy, 294 sadness.

While not enormous, this data set, as we said, covers a wide range of different topics and is useful on a broader range of sentiment and emotion classification tasks.

You can see some examples of the annotated tweets (with also the English translation) in the table below:

As you might expect, this process has a high cost in terms of annotation since we discard most of the tweets. However, it has very high precision since we collected tweets from a broad range of topics and because we have precise annotations.

Fine-Tuning (Um)BERT(o)

As standard in these recent pre-training times, we fine-tuned a BERT model with our proposed data set. BERT is one of the most popular neural architectures in Natural Language Processing. Fine-tuning BERT allows us to have a robust classification model to predict our labels. Fine-tuning is the operation that allows us to adjust the weights of the BERT model to perform our classification task.

There are many different BERT models for many languages (see Nozza et al., 2020, for a review and BERTLang). Here, we use UmBERTo, a very efficient Italian BERT model. In particular, we fine-tuned the UmBERTo model trained on the Common Crawl data set.

Sentiment Classification

To evaluate our data set and model for Sentiment Analysis, we compared our FEEL-IT UmBERTo to the same model on another data set: SentiPolc16. Both models are fine-tuned on the respective data.

SentiPolc comes with a training set and a test set. Thus, we evaluate the following interesting research question: is it better to use FEEL-IT or SentiPolc16?

We fine-tune on FEEL-IT and test on SentiPolc’s test set and compare it with fine-tuning on SentiPolc’s training set and testing on SentiPolc’s data set. Note that the model that uses SentiPolc in the training set should have a big advantage since we expect training and test to be similar. In the following picture, you can see the results.

However, these results show that using FEEL-IT is much better than using the previous state-of-the-art data set, SentiPolc.

Emotion Classification

Now, there is no data set for emotion classification on Twitter outside our FEEL-IT. Thus, we ran a simple cross-validation, showing that a fine-tuned UmBERTo model can get us to 0.71 Macro-F1! That’s an excellent result!

Creating a Python Library: FEEL-IT

One of my passion is writing code, and I try to make libraries that other people can use. Thus, Debora and I have been working on a little library the wraps the HuggingFace internal APIs to provide a simple interface for emotion and sentiment prediction.

The good thing about the library is that it allows you to do quick emotion and sentiment classification with very low coding effort: you can load your pandas dataframe, extract the text, pass it to the library, and get the results. And this is precisely what we are going to do right now.

First things first: pip install -U feel-it in your command line 🙂

Here’s the small file we are going to play with:

Now, let’s import our classifiers. The wrapper is going to take care of the downloads of the models for you.

Now, let’s get the predictions! We just need to use the prediction method of the classifier we are interested in. The classifiers take as input a list of sentences – which in this case, we will get from the CSV file I have shown before.

Do you see? With the library we have built, it is super easy to do sentiment and emotion classification in Italian!

References

Nozza, D., Bianchi, F., & Hovy, D. (2020). What the [MASK]? Making Sense of Language-Specific BERT Models. arXiv preprint arXiv:2003.02912.

Bianchi, F., Nozza, D., & Hovy, D. (2021). FEEL-IT: Emotion and Sentiment Classification for the Italian Language. 11th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis.

Sprugnoli, R. (2020). MultiEmotions-it: A new dataset for opinion polarity and emotion analysis for Italian. In 7th Italian Conference on Computational Linguistics, CLiC-it 2020 (pp. 402–408).

Acknowledgements

Many thanks to Debora and Dirk for the comments on this!