Nowadays, millions of news articles and blogs are published online every day 1. News data is created at the rate one cannot imagine a few years ago. Social media platforms have become the main source of news online to meet the information consumption needs of internet users [2]. However, a large amount of news with repeated, duplicated or junk contents is also created. How to effectively pre-process, organize and analyse the contents of the massive news is a cornerstone in providing good user reading experiences.
Therefore, there is an increasing need for automatic grouping of the news based on the contents of articles, i.e., content-driven grouping. This article explores the NLP techniques for clustering online news and extracting the trending stories over time. Typically, there are several tasks to get the top trending stories from massive news. The data processing pipeline (except news crawling) is illustrated in the following figure:
The following sections will explain all these tasks and the approach in detail. All the codes for this solution are available in my GitHub repository.
News sources and crawling
It is often necessary to collect data for text analysis from internet resources. Scrapy is a popular tool to build web scrapers. There are several interesting articles [4, 5] about using Scrapy to crawl news or related data. However, news data crawling is not the major focus here.
This article uses the financial news headline dataset from Kaggle [6] as an example to illustrate news clustering and trending story extraction. The dataset has more than 30k news headlines from the year 2018 to 2020. Each article contains the title, a short description and the publishing time. There are three news sources in the dataset, i.e., Reuters, The Guardian and CNBC.
News clean
Online news often contains many unwanted texts, words from other languages, provider-specific patterns, etc. Those text features, if not cleaned during the early stage of the pipeline, may cause noises to downstream tasks.
Text cleaning is often a domain or problem-specific task. The majority of articles in this dataset are written in English. The non-English characters are removed simply.
Provider-specific text patterns
A news provider may have some patterns in its articles. For example, the Reuters news in this dataset has many articles with common patterns of following phrases or entities.
- Facebook's Zuckerberg to testify before Congress: source
- McDonald's accused of firing worker who sued over COVID-19 claims: Bloomberg
- Coty to appoint Chairman Peter Harf as its new CEO: WSJ
- Siemens prepares for COVID-19 trough to last 6–9 months: CNBCIf those pattern phrases are not removed, they may be recognized as the keywords of the article, thereby leading to more noises to story clustering.
In this article, those provider related patterns are cleaned using regular expressions in Python.
Keywords extraction
Keywords extraction is one of the major tasks in the pipeline. The objective is to select several keywords with term frequency to reflect the key information of the article. The keywords are often from the named entities and noun phrases in the article.
Spacy is an open-source Python library, capable of most NLP applications. Using its pre-trained model, it supports fast NER (Named Entity Extraction), including most entities such as Persons, Organizations, GPE (countries, cities, states), etc. It also supports the extraction of noun phrases.
Instead of spending much effort on training a deep learning model, this article uses Spacy as the base tool to extract the entities and noun chunks efficiently.
To improve the performance of keyword extraction, the following modules are presented:
Keywords Scoring
Considering the importance of a keyword in the news, different weights are used depending on the keyword type (Entity or Noun Chunk), keyword location (title or content) and appearing times.
A keyword appearing in the title is assigned with more weight than that in the content. An Entity keyword weighs more than a Noun Chunk in the same location. A simple formula to calculate the score of a keyword is as follows:
Keywords Filtering
As machine learning models are not perfect, there are also a small number of misclassification errors in the Spacy NLP model. The objective of keywords filtering is to remove the unwanted words, misclassified entities, and symbols, such as stop words, date-times, month names, prepositions, adjectives, determiners, conjunctions, punctuations, emails, special symbols #, @, ^, *, etc.
In addition, the keywords which are related to news providers’ names and writing patterns are also removed, e.g., Reuters, Thomson Reuters, CNBC, source, story, Reuters story, etc.
Keywords postprocessing
The objective of this step is to link the entities with their alternative names or abbreviations to improve keywords similarity calculation in the next stage. There are Python libraries such as BLINK [7] to verify the extracted entities with Wikipedia data for general purpose applications.
For this 30k+ article dataset, there are about 250k keywords extracted in total. The most popular keywords are shown below.
The top 100 keywords account for about 24% of all extracted keywords, while the top 500 are about 40%. For simplicity, this article builds an entity linking table by a quick check of those top keywords.
As shown in the figure, the highlighted keywords in the same colour are referred to as the same entity. Subsequently, a simple lookup table is created to link those top keywords to their alternative names or abbreviations.
The keyword extraction in this approach runs quite fast. With a 2.5GHz CPU and 8GB RAM PC, it took about 50 minutes to complete all 30k+ news articles. On average, it needs less than 0.1s to process one article.
Story clustering
With the weighted keywords extracted for all articles, the next step is to cluster the news into stories. News story clustering is different from conventional text classification. For text classification, supervised learning methods are often used to classify text documents into a predefined list of topics or categories [8]. Story clustering is unsupervised learning [9]. Furthermore, for clustering stories from massive news, there is mostly no fixed number of clusters [9, 10]. The KMeans clustering method is not suitable for this application.
The following subsections explain the story clustering in this approach:
Keywords vectorization
This step is to convert the keywords of articles into numerical representation. The vectorizer in this article considers the scores of the extracted keywords as the term frequency. Depending on the number of news articles to be clustered, the CountVectorizer or HashingVectorizer from the Sklearn library is used. HashingVectorizer is more efficient for larger datasets.
News Similarity based on Keywords
After vectorizing the extracted keywords, cosine similarity is calculated to measure the similarity between the keywords from two articles. The similarity matrix is then constructed for the list of articles to be clustered. The codes for keywords vectorization and similarity calculation are as follows:
News Clustering
As mentioned previously, news clustering is unsupervised learning. There is mostly no predetermined number of stories for clustering massive news. In addition, the stories clustered from news may be subjective depending on the reader’s view.
For example, the below listed news articles are all related to trade war. However, if we read carefully, there are trade war stories among different countries. Even the US-China trade war story may have "sub-stories" discussing different things or impacts.
- Mnuchin says U.S. won't start a trade war: Brazil's Meirelles
- G20 talks on trade 'constructive,' no concern of trade war: Argentina
- U.S., Mexico resume talks to avert tariffs as deadline approaches
- Trade war, tariffs pose risks to U.S. and global growth: IMF, Fed officials
- EU readies new trade retaliation list before Trump visit
- USTR proposes $4 billion in potential additional tariffs over EU aircraft subsidies
- China announces new tariff waivers for some U.S. imports
- Trump prepares for 'productive' talks with Xi on trade war
- Factbox: Winners and losers in Trump's trade war with China
- China to slap additional tariffs on $16 billion of U.S. goods
- China rare earth prices soar on their potential role in trade war
- Oil prices slide as U.S.-China trade war escalatesReaders may have different views regarding the stories in news. Therefore, there is no clear cut answer on the number of stories. In this article, DBSCAN method is used to cluster news articles.
Parameter eps is the maximum keyword distance (1-keyword similarity) of two news articles in the same cluster. When eps approaches 0, the clusters are more cohesive at the risk of clustering the same story news into different clusters. When eps is 1, all news articles fall into the same cluster. Here, eps is set 0.35 initially.
In addition, another parameter _maxsize of the cluster is introduced. For the reason that there are many articles with the top keywords (such as US, China, Trump, etc), those articles may form super-sized clusters. The algorithm will further cluster the news in the superclusters using smaller eps until all cluster sizes are less than _maxsize.
The code for this clustering algorithm is available in my git repository here.
Trending stories visualization
After news articles are clustered, the trending stories are extracted straightforwardly. For any given date, the trending stories here are defined as the top clusters with more news articles published in the past certain time period, e.g., the past two weeks.
The top trending stories between Nov 2019 and July 2020 extracted from this dataset are visualized in the following figure.
In this figure, the circle size is calculated by taking the logarithms of the number of articles in the cluster. The keywords in the circles are the keywords for those top stories in a particular time period.
Conclusions and Discussion
This article presented a Python solution on how to extract the trending stories from massive news. The solution covers data cleaning, keywords extraction, news clustering, and story visualization.
Although this solution was presented for this financial news application, it is not difficult to extend to other domain datasets by minor customization on data cleaning, keywords filtering and the entity linking table.
In addition, the keywords module was developed using the Spacy library with its pre-trained model. This solution focuses more on efficient keywords extraction and news clustering, rather than a deep learning approach requiring much training efforts.
Further improvements may be done in the areas of domain-specific knowledge graphs for better entity linking, content-driven clustering, etc.
References
[2]: How Social Media Has Changed How We Consume News, https://www.forbes.com/sites/nicolemartin1/2018/11/30/how-social-media-has-changed-how-we-consume-news/?sh=6d403f743c3c
[3]: Tom Nicholls & Jonathan Bright (2019) Understanding News Story Chains using Information Retrieval and Network Clustering Techniques, Communication Methods and Measures, 13:1, 43–59, DOI: 10.1080/19312458.2018.1536972
[4]: Web Scraping news articles in Python, https://towardsdatascience.com/web-scraping-news-articles-in-python-9dd605799558
[5]: Using Scrapy to Build your Own Dataset, https://towardsdatascience.com/using-scrapy-to-build-your-own-dataset-64ea2d7d4673
[6]: Financial News Headlines Data, https://www.kaggle.com/notlucasp/financial-news-headlines
[8]: Text Classification in Python, https://towardsdatascience.com/text-classification-in-python-dd95d264c802
[9]: M. Tarik Altuncu, Sophia N. Yaliraki, and Mauricio Barahona. 2018. Content-driven, unsupervised clustering of news articles through multiscale graph partitioning. In Proceedings of 2018 KDD Data Science, Journalism and Media
[10]: Identifying hidden trends in news stories using hierarchical clustering, https://towardsdatascience.com/identifying-hidden-trends-in-news-stories-using-hierarchical-clustering-b6297df795af
