An example of how linguistic domain knowledge can inform feature selection for insight.

What makes a movie review negative or positive? I mean, obviously a real stinker of a movie ("Oh Hi Mark") is going to produce some clearly critical reviews, but how do we as humans read a review and understand the reviewer’s take in general? This is an important question in machine learning (ML), in part because improving the performance of ML may hinge on better understanding of how we as humans do things like read a review. Now, accurately classifying movie reviews as thumbs up/thumbs down is pretty simple (existing methods are already about 99% accurate). Still, teaching machines to read more like humans could mean classifying movie reviews with more granularity, or perhaps move us towards machines that can make sense of complex, higher stakes things (like corporate ethics statements or online conspiracy theories).
Note: plenty of people in ML don’t think we need to understand the "how" of human tasks like reading. They argue we need better algorithms, more/better compute power, and (more than anything) more data. Regardless, I’m still interested in answering "how" questions like this, partly because I’m a scientist: I want insight that could help solve problems. But also partly because I’m a scientist: I just want to know how things work.
Words like "stinker" are a first clue to how we read reviews: words that are not context-dependent, but consistently have a good/bad meaning. This meaning (semantics) is directly about assessments of negativity or positivity, and so if someone uses a lot of obvious positive words ("wonderful," "joyous," "delight"), that’s a darn good clue that we are reading a positive review. We can see this using an "oldy but goody" ML algorithm: the Random Forest (RF) classifier. The RF classifier is nice here because in addition to generally performing well, it can output feature importance: what features of the data did it use to learn to predict (whatever)?
Using Words to Understand Movie Reviews
We can start off by using words (the lexical level of language) for our RF classifier. In this first attempt, we’ll score every movie review by it’s term frequency over inverse document frequency, or TF-IDF. In essence, this means training the classifier by teaching it to look for terms that are frequent in some documents, but infrequent in other documents. It’s easy to undertsand if you think about the sections of a traditional newspaper: words like "then," or "also" are going to be common to every article in the paper. But words like "financial," "markets," "rates" and "economy" while frequent for some articles (i.e. the business section) are relatively infrequent in other sections.
When we feed in our training data (text from thousands of IMDb reviews), labeled by human readers as either "Neg" or "Pos," and then broken into words/word pairs into our RF classifier, the machine does a pretty good job figuring out how to tell a negative from positive review:

You can read the above confusion matrix across the diagonal from left to right: about 86.5% correct predicting negative reviews, and about 84.4% accurate for positive reviews. Then to understand how the model makes predictions, we can output the feature importance list:

No big surprise here: words like "bad," "worst" and "awful" are important for identifying negative reviews, and corresponding words like "great," "excellent," and "best" for positive reviews. There are two features that are a little different: "just" (a qualifier) and the bigram "waste time (a value judgement?). But overall the top 20 highest weighted features aren’t particularly insight producing.
Part of that may be how low the importance values are (basically a value between 0 and 1 that represents on average what percent that feature contributes to the model). For example the single most important feature ("bad") has a paltry 0.0146 importance value. That may be a clue to why there isn’t a very discernible pattern to the feature importance list beyond simple good/bad: maybe there are so many different words that contribute to a review’s sentiment that no one of them has much weight. It’s hard to see a pattern among thousands and thousands of words. Maybe we need something more sparse to get insight.
Linguistic Stance: a Taxonomy of Language Moves
Instead of using words and pairs of words, what if we use categories of words as our features? David Kaufer and Suguru Ishizaki of Carnegie Mellon University have produced a useful taxonomy of linguistic stance in their DocuScope project. DocuScope is built on an expert dictionary derived from functional Linguistics and rhetoric, and covers many categories of words, e.g. certainty/uncertainty, emotion, cultural values, social relationships, and so on. These are the various linguistic stances we adopt in speaking/writing. While there are millions of words and phrases in the dictionary, there are relatively few stance categories (119 in the version used here). So instead of passing thousands of words to the RF classifier, we can use DocuScope to count up the number of times any stance category appears in our document: X% negativity words, Y% abstract concepts, Z% argument/denial, and so on. When we pass those vectors on (essentiually a list of 119 stance values per review) to the RF classifier, an interesting trade-off happens.
Model Accuracy vs Interpretability
Using stance categories (the lexicogramatical level of language) as features doesn’t improve performance over a word-based model:

Again reading diagonally from left to right, the stance-based model is about 80% accurate predicting positive reviews, and around 73% accurate with negative reviews. So from a pure performance perspective, using stance hurts performance. But insight is a very different story:

The two most important features are old friends – generally negative and postive words – no surprise there. But they are a stronger signal now: where the single word "bad" explained roughly 1.5% of feature importance importance, the stance category "negativity" explains 9%.
It gets more interesting from there. Two values categories show up next, one about positive values and one about negative value, both in the public sphere. These words refer to social goods (e.g. peace, justice) and social ills (e.g. corruption, poverty), something more complex than simple good/bad polarity. It turns out that positive reviews include talk about "our progress as a species," or address important subjects more "educationally" and in a "more effective way" than bad films. Conversely, crummy films "cross the line," are marked by "gratuitous violence," and are "unimaginative, implausible and convoluted" in their plots. So beyond simple polarity, cultural values are part of teaching a machine to read more with more sophistication. Opposition argument langauge ("DenyDisclaim" above) is also important. For example in a negative review, you might say that if you "had not" read the book, you might have enjoyed the film, but "unfortunately this is not the case."
The stance-based model tells us that understanding movie reviews most strongly hinges on language moves around positivity/negativity, public values, argument opposition, abstract concepts, and social identity ("PersonProperties" in the above list). Some of that (positivity/negativity) we could have guessed (although it is nice to have this empirically confirmed and quantified), but the other parts we might not have guessed. This is an example of the many things humans can do – be ironic, persuasive – but that are harder to unpack analytically. To do that, we may benefit from a distinct reading perspective: machine-reading.
Different Ways of Seeing the Same Thing
Because computers and humans read differently, they can be highly complementary. Humans read with lots of context, bringing in outside knowledge to make richer and more precise meaning of text data. But we are also slow, while computers reading without context read at superhuman scale/speed. Also, human precision in reading is serial: we read one thing at a time and can’t see the whole. Whereas machines have a bird’s eye view of a large text dataset. In many dimension, machines can complement human reading.
Researchers at Stanford’s Literary Lab have compared human and machine reading using the analogy of identifying writing genres as if they were buildings. Human beings can use themes (dark secrets, mounting dread) to identify a genre like gothic fiction, just as people can use architectural elements (plinths, smooth surfaces) to identify an architectural genre like neoclassical. Whereas computers can use word counts (lexical level features) and DocuScope’s stance categories (lexicogrammatical features) to identify that same genre of gothic fiction. It’s something like using the grains of sand (words) in the mortar or type of bricks (stance) to identify a building architecturally. Of course, it’s not that humans don’t use words or stance to read. Rather, we make use of them without seeing larger patterns explicitly, the way we recognize a pattern like Art Deco without being able to easily rattle off the micro features that build up to the pattern.
I really like the idea of giving machines additional kinds/representations of data to help them read better. Then in turn, being able to look under the hood and understand how machines leveraged those additional representations in reading text data, because it can expand human knowledge, and likely for things more important than movie reviews.