Hands-on Tutorials
Text-based Causal Inference
Tutorial on analyzing voter fraud disinformation by estimating causal effect with text as treatment and confounder
Science fiction tells us that rampant disinformation is a foresign of a society’s descent into a dystopia. It could be argued that disinformation destabilizes a democracy (Morgan 2018, Farkas & Schou 2019). Tangibly, people disregarding medical evidence has a negative impact on public health. For instance, people who are willing to ignore evidence might choose to refuse vaccines and thereby endanger others’ lives and their own. One should be cautious because scientific disinformation is pervasive, but it is hard to hold people accountable when access to trustworthy news material is garbled by the influx of fake news. A more insidious form of disinformation is the subversion of reality for a subset of the population; a type of mass hysteria that captures those cognitively vulnerable in an alternative reality. I am speaking of the surreal claims of voter fraud over the last American election, when Trump’s sycophants refused to accept that he lost the election. The fake news around voter fraud had an undeniably incendiary impact on the following January 6th insurrection, a tragic event that screamed dystopian society.
Hannah Arendt, the political philosopher, claimed it was necessary for people to engage with politics as a part of living a good life. In “The Human Condition”, Arendt says that it is not enough to work and spend time with those you love, you must also engage in political life (Arendt, 1958). There are many Americans who follow this ethos and engage with politics; viewing it as their right and responsibility as a citizen. Unfortunately, some of them are susceptible to fallacious thinking and fall prey to bizarre conspiracies like QAnon. In “Networked Propaganda” by Benkler, Faris and Roberts, the authors claim that the propaganda feedback loop is partly fueled by people’s desire to avoid cognitive discomfort. That is to say, people will seek out information that reinforces their worldview while ignoring or discounting evidence to the contrary. In an epidemiological sense, fake news acts as a vector of disease, spreading dangerous disinformation and saturating the public sphere with conflicting accounts that make it nigh impossible to discern the truth.
But what does politics have to do with data science? As a researcher interested in disinformation, I naturally seek to use data science tools to answer questions of social and political interest. Of immediate interest, is understanding the relationship between social media and fake news. There are claims that the toxic nature of social media, which is driven by shock-value and upvoting, has an impact on the dissemination of fake news. More specifically, looking at Twitter, I question whether fake news has a causal impact on the outcome of retweet count. Does sharing fake news lead to a higher retweet count? This tutorial is a result of my attempt to answer that question, and is a follow up to a previous article about causal inference using NLP.
In this article, to estimate causal effects using text, I make use of the Twitter VoterFraud2020 dataset curated by the Jacobs Technion — Cornell Institute. This dataset was made publicly available by the researchers and shared on this dashboard, the original paper is attributed to Abilov et al. (2021). I start by discussing the data and describing the preliminary analysis. Next, I introduce the causal-text algorithim and walkthrough the corresponding study on causal effects of linguistic properties (Pryzant, 2021). Additionally, I cover guidelines for using observational data for causal inference, and detail the estimation procedure of the causal-text algorithm. Following that, I lay out the framework for the causal experiment, which leads directly into a tutorial of how to set up and use the causal-text tool (which I fork and adapt from the original repo). I also cover the steps required to address the proposed causal question with the causal-text algorithm. Lastly, I briefly discuss the results and consider possible extensions.
Data Description
The open source Cornell VoterFraud2020 Twitter dataset contains 7.6M tweets and 25.6M retweets from 2.6M users, all related to the claims of voter fraud between October 23, 2020 to December 16, 2020. Due to Twitter’s privacy policies only the tweet ids and user ids are shared; however, the GitHub repository for the dataset includes scripts to hydrate the data. For this experiment, I focused only on the 7.6M original tweets. Once the tweets were collected, it was necessary to do some preprocessing to clean the tweet text and extract urls. The urls were all in Twitter’s shortened format of “t.co”, and thus had to be resolved. To better understand the popularity of the resolved urls, each url was given an Alexa rank from Amazon’s analysis of web traffic statistics.
Media Cloud is an open-source media content analysis tool developed at the Berkman Klein Center for Internet & Society at Harvard University. This platform has curated source lists for American media sources, divided by political affiliation, covering the left, center-left, center, center-right and right. Using these US news source lists, I cross referenced with the urls resolved from the voter fraud tweets dataset. This selected for urls that linked specifically to news articles. Media Cloud has great investigative capabilities for news media, so I was able to use Media Cloud queries to determine the media in-link share count and Facebook share count of the isolated articles. In addition to this article metadata, I scraped the full text of all the news articles that were still online at the time.
These steps gave me a combined dataset of original tweets that shared news articles, with the full text of the news articles, article metadata, tweet metadata and Alexa rankings for the urls. To be clear, the dataset of 7.6M tweets was pared down such that every tweet had a corresponding news article. The purpose of collecting the full text of the articles was to perform topic modeling with latent Dirichlet allocation (LDA) to see if it was possible to isolate the fake news articles. Additionally, the VoterFraud2020 dataset of 2.6M users also contained each user’s community or cluster as determined by a community detection algorithm (e.g. Louvain method). Given the multiple data streams and richness of the resulting tweet-article dataset it was necessary to run some preliminary analysis, which is covered next.
Preliminary Analysis
Firstly, considering the tweets themselves, the text of the tweets may have value in identifying fake news. Hence, I started by topic modeling the tweet text with LDA to gain a general sense of the content of the Twitter discourse.
The results of the LDA model highlighted several topics that were expressly about aspects of the voter fraud disinformation conversation. For instance, one notable topic that was isolated from the other topics, was about the alt-right hashtag “#stopthesteal”. Also notable was the similarly isolated Wall Street Journal and New York Times fact checking topic. Interestingly, tweets that claimed to have evidence of fraud greatly overlapped with tweets from right-wing news media sources, like Fox News. Overall, there are several threads of disinformation, ranging from ballot harvesting, conspiracies about the voting software, affidavits asserting fraud, and rumours of military involvement.
Analysis of the user communities with a community detection algorithm provided 5 distinct communities that varied in user count, as shown below.
The amplifier groups were those that pushed the voter fraud agenda, and the foreign groups represent potential foreign influence and are tiny in comparison.
Focusing on the spread of fake news, these five communities behaved differently when it came to sharing urls of news articles. The Media Cloud metadata contained the media inlink count which represents the number of times an article was linked to by other media sources. A highly in-linked article could be considered to be more mainstream. The graph below shows the trend of media in-link count over time for the three largest communities.
The above time series suggests that the center-left community tended to share articles that were more “mainstream” when compared to the two amplifier communities. Although the center-left community was more likely to share mainstream articles, they did not garner high retweet counts. This is shown below in a time series of mean retweet counts by community.
In fact, it is the third largest community “Amplifiers_1” that had the highest retweet counts for shared urls, despite representing only 11.5% of users. The concern here is that even if the center-left were trying to fact check those spreading the fake news of voter fraud, they were not getting much visibility on Twitter. It is also startling to realize that the relatively small group of “Amplifiers_1” was highly influential in spreading information, despite not sharing mainstream media.
It is commonly assumed that fake news is often on fringe websites, far out of the mainstream. Having calculated the Alexa rank or popularity of each news article url, it was possible to look at the relationship between a website’s “fringiness” and which communities were retweeting these fringe sites. In the heatmap below, the Alexa rank or “fringe-score” is weighted by retweet count and the community distribution by topic is mapped.
Here we can see that the “Amplifier_1” group not only garners the largest retweet count shares, but they also share the most fringe websites. Since we are interested in the causal question of whether the treatment of fake news has a causal effect on the outcome of retweet count, the relationship between fringe-scores and fake news articles is also of interest.
At this point, it became necessary to look at the actual text of the news articles to better help classify fake news. The process of topic modeling the news articles with LDA resulted in five of the seven topics being explicitly fake news articles. This allowed for labeling each url, and thereby each tweet, as either fake news or not. This NLP derived label is used as a proxy label in the setup of the causal experiment that is described later. Furthermore, in order to test the usefulness of the causal-text algorithm, I also labeled 100 of the most popular urls for fake news. This labeling covered 18% of the tweet-article dataset, and gave me roughly 28K tweet-article pairs that had both proxy labels via topic modeling and true labels via manual annotation. Having both proxy treatment labels and true treatment labels, allows for benchmarking the causal-text algorithm for this task. In the next three sections, I will discuss the details of the causal-text algorithm and introduce some of the causal concepts needed to understand the tool.
Causal-text Algorithm
The causal-text algorithm used in this tutorial was created by Pryzant et al. (2021), it was introduced as “TEXTCAUSE” in a paper titled “Causal Effects of Linguistic Properties”. This causal algorithm makes use of another tool — CausalBERT, that was originally designed by Veitch et al. (2020). CausalBERT was developed to produce text embeddings for causal inference; in essence, the authors designed a way to use AI language models to adjust for text when testing for causality.
The causal-text algorithm has two components, first it makes use of distant supervision to improve the quality of proxy labels, and second, CausalBERT is used to adjust for the text. Pryzant et al. tried to formalize the causal effect of a writer’s intent, along with establishing the assumptions necessary to identify the causal effect from observational data. Another contribution of this work is that they proposed an estimator for this setting where the bias is bounded when adjusting for text.
The VoterFraud2020 dataset represents observational data, where the tweets were obtained without an intervention. Since, the measurement of causal effect requires the fulfillment of the ceteris paribus assumption, where all covariates are kept fixed, we must reason about interventions. Pryzant et al. describe two challenges to estimating causal effects from observational data. Firstly, there is a need to “formalize the causal effect of interest by specifying the hypothetical intervention to which it corresponds.” (Pryzant et al., 2021). This challenge is overcome by imagining an intervention on the writer of a text, where they are told to use a different linguistic property.
The second challenge to causal inference is identification, where the actual linguistic property we are interested in can only be measured by a noisy proxy (e.g. topic labels). Therefore, the study also established the assumptions needed to recover the true causal effects of linguistic properties from the noisy proxy labels. The creators of the causal-text algorithm adjust for confounding in text with CausalBERT and prove that this process bounds the bias of the causal estimates. In my previous article about causality and NLP, I discussed in detail the issue of confounding due to text.
Causal Inference with Observational Data
When discussing causal inference with observational data, it is necessary to talk about the average treatment effect (ATE). As seen in the image below, the ATE is the difference in potential outcomes between the real world (T=1) and the counterfactual world (T=0). I have previously described the potential outcomes framework in intuitive ways in two articles: Causal inference using NLP, and CausalML for Econometrics: Causal Forests.
However, as mentioned, we are also concerned with confounding. To deal with confounders (Wᵢ), the backdoor adjustment formula (Pearl, 2009) can be used to rewrite the ATE in terms of all the observed variables: Tᵢ for treatment and Yᵢ for outcome. This confounding relationship is seen in the image below, where the confounder Wᵢ has an effect on both the treatment and outcome.
The confounding effect of Wᵢ results in a spurious correlation, which can also be referred to as “open backdoor paths that induce non-causal associations” (Pryzant et al., 2021). I have previously discussed spurious correlations and backdoor paths in an article about improving NLP models with causality. The backdoor adjustment formula for the ATE is shown in the image below.
If we assume the confounder Wᵢ is discrete, then the data can be grouped into values of W, the average difference in potential outcomes can be calculated, and lastly, we take the average over the groups of W.
Pryzant et al. (2021) propose the following causal model of text and outcomes:
The text is represented by W, which has the linguistic property T (as treatment), and other qualities Z (as covariates). Here, Z could be the topic, sentiment, length or other qualities of a text. This causal model is built with the literary argument that language is subject to two perspectives: the text as intended by the author and the text as interpreted by the reader. The second perspective of the reader is shown by T_tilde and Z_tilde — where T_tilde represents the treatment as received by the reader, and Z_tilde represents the other qualities of the text W as perceived by the reader. The outcome Y is affected by the tilde variables instead of Z and T directly. The T_hat variable represents the proxy label as derived from the text W, which could be topic labels.
The hypothetical intervention on the treatment, is to ask the writer to use (or not use) a linguistic property T, where T is a binary choice. It is not possible to use observational data to capture the unobserved linguistic characteristics of Z, because it is correlated with T. It is, however, possible to estimate the linguistic properties as perceived by a reader, which is represented by the tilde variables. The ATE of the reader’s perspective is defined as:
In order to calculate the causal effect of interest, the ATE of the writer’s perspective, Pryzant et al. (2021) developed a theorem (Theorem 1) which exploited the ATE of the reader’s perspective as calculated from T_tilde. They define Z_tilde as a function of the text W, as seen in the image below, where the potential outcomes of Y are equivalent given either W or both T_tilde and Z_tilde.
Having defined Z_tilde as such, it is then possible to define the ATEᵣₑₐ as the following equation:
The ATEᵣₑₐ is said to be equal to the ATE𝓌ᵣᵢ, and the text W, splits into the information that the reader uses to perceive the tilde variables. Z_tilde represents confounding properties, since it affects the outcome and is correlated with T_tilde. To be clear, this theorem only holds under certain assumptions, of which there are three. Firstly, unobserved confounding (W) blocks the backdoor paths between T_tilde and the outcome Y. Secondly, we need to assume that T = T_tilde, that is to say, there is an agreement of intent (ATE𝓌ᵣᵢ) and perception (ATEᵣₑₐ). The last assumption is the positivity (or overlap) assumption, which is that the probability of the treatment is between 0 and 1. I provided an intuitive explanation of the positivity assumption in another article about causality.
A further complication is that we cannot observe the perception of the reader, in addition to not being able to directly observe the intent of the writer; hence, the need for proxies. For T_tilde it is possible to use a proxy T_hat to calculate the causal effect of interest, where the T_tilde is substituted out by T_hat in the previous equation to calculate an estimand (ATEₚᵣₒₓᵧ).
At this point it is necessary to adjust the estimand for confounding, in other words, adjust the ATEₚᵣₒₓᵧ for Z_tilde. This is made possible using CausalBERT, a pre-trained language model, to measure T_hat. The other advantage of this approach is that the bias due to the proxy label is bounded, such that it is benign — “it can only decrease the magnitude to the effect but it will not change the sign.”. Pryzant et al. (2021), refer to this as Theorem 2, and state that a “more accurate proxy will yield a lower estimation bias.”.
Causal Estimation
Now that we have discussed how to use observational data for causal inference with text, the practical part is the estimation procedure. The causal-text algorithm has two important features: improving proxy labels and adjusting for text. The approach to improving the accuracy of the proxy labels is based on the fact that the bias is bounded. The proxy labels are improved using distant supervision, which was inspired by work on lexicon induction and label propagation. The goal is to improve the recall of proxy labels by training a classifier to predict the proxy label, then using that classifier to relabel examples that were labeled T=0 but look like T=1. Essentially, the proxy labels are relabeled if necessary.
The second feature of the causal-text algorithm is that it adjusts for the text using a pre-trained language model. The ATEₚᵣₒₓᵧ is measured by using the text (W), the improved proxy labels (T_hat*) and the outcomes (Y). This relies on Theorem 1, which as described earlier shows how to adjust for the confounding parts of text. Pryzant et al. (2021) use a DistilBERT model to produce a representation of the text with embeddings and then select the vector corresponding to a prepended classification token, [CLS]. Pryzant et al. use Huggingface transformers implementation of DistilBERT which has 66M parameters, and the vectors for text adjustment Mₜ , add 3,080 parameters. This model is then optimized such that the representation b(W), directly approximates the confounding information, Z_tilde. An estimator, Q, is trained for the expected conditional outcome, as seen in the image below.
In this equation, the estimator Q is shown to be equivalent to the expected conditional outcome for Y, when given the proxy T_hat, which itself is based on not only the treatment, t, but also the model representation of Z_tilde (b(W)), and the covariates C. The proxy estimator, Q_hat, is equivalent to the parameterized sum of a bias term (b) and two vectors (Mᵇₜ , Mᶜₜ ), that rely on representation b(W) and a c vector. The c vector is a one-hot encoding vector of covariates C, and the two Mₜ are learned for value t of the treatment. The training objective of this model is to optimize:
In this equation, 𝛩 is all the parameters of the model, and L(.) is the cross-entropy loss which is used with the estimator Q_hat, itself based on the Mₜ vectors. The original BERT masked language modeling objective (MLM) is represented as R(.), and the 𝛼 hyperparameter is a penalty for the MLM objective. With the Q_hat estimator, the parameters Mᵇₜ and Mᶜₜ are updated on examples where the improved proxy label is equivalent to t.
This setup is shown in the diagram below, where W represents the text, C represents the covariates and the CausalBERT model is a representation of the text such that it is possible to predict the potential outcomes of Y.
In summary, estimation with the full causal-text algorithm requires the improved proxy labels, and a causal model of the text and outcomes which extracts and adjusts for the confounding of Z_tilde. The algorithm also allows for the inclusion of covariates C when estimating the causal effect. As seen in the image above, the vector c and the model representation b(W) are used to predict the potential outcomes of Y while using information from T_hat*, the proxy label. The representation b(W) directly approximates the confounding information (Z_tilde), which allows it to adjust for the text.
Once the estimator Q_hat is fitted, it is possible to calculate the hatted ATEₚᵣₒₓᵧ as seen in the equation below:
The ATE that is derived with this method can be used to determine the causal effect of the reader’s perspective, which itself is assumed to be equivalent to the causal effect of the writer’s perspective. The accuracy of this ATE is dependent on how accurate the proxies are and how well CausalBERT adjusts for the text. The next section describes the experimental framework used to test for the causal effect of fake news on retweet count.
Experimental Framework
The causal question is whether fake news has a causal effect on retweet count. A few years ago, a very popular study in Science claimed that on social media, fake news spreads more rapidly than real news. This study, however, does not rely on causal analysis. There is a possibility that the results were based on spurious correlations between confounders and the retweet count. For instance, the community a user belonged to was not investigated, nor was the popularity of the news site. Some communities might be more vulnerable to spreading fake news, and people might be more likely to share popular news sites. Moreover, language is complex and the text of tweets can act as a confounder, so we need to control for topic, writing style, tone and length of the tweet. Therefore, there is value in designing a causal study where possible confounders are controlled for, and the tweet text itself is adjusted for confounding qualities.
As previously mentioned, there are two challenges to estimating causal effects from observational data: interventions and identification. Firstly, we need to reason about the hypothetical intervention that we would make on the writer’s intent, such that they would use (or not use) a particular linguistic property. It is necessary to think of the sharing of fake news as a linguistic property that represents the writer’s intent, then it can be a treatment, T, that can be intervened upon. More simply, we view the sharing of a url that links to fake news as a linguistic property, where the intervention would be to tell the user to share a real news article (T=0) instead of a fake news article (T=1). While intervening on this treatment, the remaining qualities of the tweet must be kept constant. We will refer to these other text qualities as Z, such that Z represents potential confounding factors like topic, style, tone or length. The tweet text will be referred to as W (or simply “text”), and additional covariates such as user community or Alexa rank will be denoted as C. This setup is shown in the image below.
The proxy treatment is a fake news label as determined by topic modeling of the articles with LDA. Since the dataset has gold labels for the fake news articles, there are two treatment variables (T_true and T_proxy) so that it is possible to benchmark the T_proxy against T_true. Lastly, the outcome Y is the retweet count. For a first test, the C variable is categorical, where a number is used to represent the user community. All other variables, except the text, are binary numerical indicators (0 or 1). For a second test, the Alexa rank is used as the covariate C, and we look specifically at a single community: “Amplifiers_1”. In this test, the Alexa rank for each url is turned into a categorical variable by binning the values by quantile. The next section details how I adapted the causal-text algorithm for this tutorial, and explains how to interpret the results.
Estimating Causal Effect
Pryzant et al. (2021) shared the causal-text algorithm on GitHub, which utilized a Pytorch implementation of CausalBERT. For this tutorial, it was necessary to adapt the original causal-text package since it was specifically tailored to the causal experiments described in the introductory paper. Furthermore, it does not appear to be maintained (updated) by the authors, so I had to update the requirements. I also simplified the output and removed the extraneous simulation parts that were not needed for this tutorial. The rest of the changes were minor and were made in the process of debugging. Overall, I made very few changes to the original algorithm, my adaptation can be accessed on GitHub. If you find this algorithm useful, please star Pryzant et al.’s original repository for the causal-text algorithm.
The tool runs from the command line, and I suggest running it with a GPU to make use of the speed for deep learning. Here, I explain how to set up Colab (to make use of the free GPU instance) and run the causal-text algorithm. First though, the data needs to be in the correct format. The tool accepts a “.tsv” file with five columns, for five variables: T_proxy, T_true, C, Y, text. The covariates C have to be categorical, and represented by simple integers. The T_proxy, T_true and outcome, Y, variables must be binary numerical indicators (0 or 1). The “text” is simply the tweet text. The adapted causal-text algorithm produces seven different ATE result values.
Using the T_true label, the causal-text algorithm calculates an “oracle” ATE value; this can be thought of as the true ATE which will act as a baseline. Next, an “unadjusted” ATE value is calculated as an additional baseline, where the ATE is the expected difference in outcomes conditioned on T_hat without accounting for covariates. The next two values are “T-boost” ATE values, where T-boost refers to treatment boosting by improving proxy labels. The proxy labels are improved in two ways by two different classifiers. One classifier works only with positive and unlabeled data, while the other is a straightforward regression, specifically Sci-kit Learn’s stochastic gradient descent classifier. The next ATE value is one where the text has been adjusted for, this is the “W adjust” value. The last two ATE values combine the T-boosting with text adjustment, for one ATE value per classifier type. These last two values represent the full “TEXTCAUSE” algorithm as designed by Pryzant et al. (2021).
The first step is to install the needed packages in Colab. This is done with the following single line of code:
!pip install sklearn transformers tensorflowNext, we check to see if the GPU is available.
import torchif torch.cuda.is_available():
device = torch.device("cuda") print('There are %d GPU(s) available.' % torch.cuda.device_count())
print('We will use the GPU:', torch.cuda.get_device_name(0))
!nvidia-smielse: print('No GPU available, using the CPU instead.')
device = torch.device("cpu")
The “.tsv” file with the data should be saved in Google Drive for easy access. We simply mount the drive to get access to the files.
from google.colab import drive
drive.mount('/content/gdrive')Then we navigate to the folder where the data is saved.
%cd gdrive/My Drive/my_folderNext we clone the adapted repo for the causal-text algorithm from GitHub.
!git clone https://github.com/haayanau/causal-text.gitAfter the causal-text package has been cloned, it is necessary to navigate to the directory where the main script is located.
%cd causal-text/srcRunning the algorithm is very simple, run the following command with the path leading to the “.tsv” file. The “run_cb” argument means that CausalBERT will be used to adjust for the text. The models are trained for 3 epochs each.
!python main.py --run_cb --data /content/gdrive/MyDrive/my_folder/my_data.tsvThis command results in seven types of ATE values as described earlier. Pryzant et al.(2021) do caution that “ATE estimates lose fidelity when the proxy is less than 80% accurate”. They also claim that it is crucial to adjust for the confounding parts of text, and that estimates that account for C without adjusting for the text, might be worse than the unadjusted estimate. The next section briefly discusses the results of the two experiments, and suggests some extensions.
Results and Extensions
For the first test, we are looking at the ATE of fake news (T) on retweet count (Y), with the user community (C) and tweet text as confounders. There are 15,468 observations and the results are shown below.
The true (oracle) ATE value suggests that there is practically no causal effect, which is contrary to the popular expectation that fake news spreads quicker than real news, and would garner higher retweet counts. The unadjusted ATE value also does not show causal effect, despite not accounting for covariate C. In terms of matching the true ATE value, the (W adjust) ATE that adjusts for text with CausalBERT comes the closest. Neither of the values from the full “TEXTCAUSE” algorithm (adjusts for text and improves labels) are as close to the true ATE as the W adjust value, which does not use the improved labels.
The second test looked only at the “Amplifiers_1” community and accounts for Alexa rank as a potentially confounding covariate. There are 1,485 observations and the results are shown below.
Here again, there does not appear to be a causal effect of fake news on retweet count, once we control for the Alexa rank. The true (oracle) ATE value is mildly negative, and the full version of the “TEXTCAUSE” algorithm that uses the “pu” classifier for T-boosting, produces the ATE that most closely matches the true value. This treatment boosted value (TextCause pu), not only includes improved proxy labels but also adjusts for the text with CausalBERT. The unadjusted ATE had the worst performance, however, all of the other ATE values had a similarly poor performance.
There is a strong possibility that there are unobserved confounders that have not been controlled for in this experiment. This might explain the lack of causal effect being detected, or conversely, there simply is no causal effect. At this point, we have not proven that there is a causal effect of fake news on retweet count, nor have we definitively proven that there is no causal effect. All we have done is bring into question the assumption popularly claimed by researchers, that fake news spreads faster than real news on social media. It might be possible that including further covariates would improve the experiment, however, it will be tricky to determine which covariates to include. There is also a possibility that the sample sizes were not big enough, especially for the second test where there were only 1,485 observations.
There are several extensions that we could implement. Starting with the first test, we could substitute the user community with Alexa rank for the covariate, C. For the second test, we could increase the sample size or even compare across communities. It would be helpful if the causal-text algorithm could accommodate more than one covariate (higher dimensionality). Even more useful, would be if the causal-text algorithm could deal with heterogeneous treatment effects and calculate the conditional average treatment effect (CATE). For instance, we could condition on user community, to see if there is a difference in CATEs across the groups.
Final thoughts
The intersection of causal inference and NLP is fascinating, and the causal-text algorithm is a great example of creativity and initiative. My hope is that this research will continue to push the boundaries of what is possible with regards to methods for estimating a causal effect with text. In terms of application, the causal-text algorithm can be applied to various fields such as economics, healthcare, marketing, public policy and even epidemiology. There is a shift in thinking around the phenomenon of fake news, for instance, there are calls to treat the issue as a public health issue (Donovan, 2020). The WHO has taken an epidemiological approach, and are referring to incidents of fake news as “infodemics”. All of these changes suggest that it might be time to take a causal approach to disinformation. Exploring causality could be a way of developing an economics-inspired framework, for looking at the causal effect of disinformation on society (True Costs of Misinformation Workshop, TaSC, Shorenstein Center). Personally, I am interested in applying this method to economic research that utilizes open source social media data.
I welcome questions and feedback, please feel free to connect with me on Linkedin.
