Explorations in Named Entity Recognition, and was Eleanor Roosevelt right?
Using the spaCy Natural Language Processing lib to gain insight from news articles
Eleanor Roosevelt is alleged to have said:
Great minds discuss ideas; average minds discuss events; small minds discuss people.
And although this might be a misattribution, the statement as such seems to resonate with a lot of people’s intuition, but how true is it? Does it stand up to scrutiny?
There are many ways in which this could be investigated, one fun approach might be to look through a bunch of newspapers for ideas, events and people and see if the fraction in which they appear can be correlated to the “mind size” (great, average, small) of its readers.
To mine the newspaper articles for information, I decided to use a natural language processing technique called Named Entity Recognition (NER), which is used to identify something called “named entities” in a sentence. Named entities are things such as products, countries, companies, numbers. I will use the spaCy natural language processing lib for this. Here’s an example from their documentation of how NER-tagging can look:
