
This article is authored by Ayaan Haque and Viraaj Reddi
In this article, we will review our recent work titled "Deep Learning for Suicide and Depression Identification with Unsupervised Label Correction" by Ayaan Haque, Viraaj Reddi, and Tyler Giallanza. This paper is a method that uses deep learning for suicide vs depression identification in the presence of noisy labels, and is currently under review at a top conference. Our paper is available on ArXiv, the code is available on Github, and the project website is here. This article will review our method, titled SDCNL, as well as some of the results.
Overview
SDCNL, which stands for Suicide Depression Classification with Noisy Labels, is a method for distinguishing between suicide and depression using deep learning and noisy label correction. Early detection of suicidal ideation in depressed individuals can allow for adequate medical attention and support, which in many cases is life-saving. Recent NLP research focuses on classifying, from a given piece of text, if an individual is suicidal or clinically healthy. However, there have been no major attempts to differentiate between depression and suicidal ideation, which is an important clinical challenge. In addition, due to a lack of data availability, we use data from online forums such as Reddit to train our model. However, unlike other sources of data, data from online forums have not been verified as true and accurate. Thus, this introduces the issue of noisy labels **** from web-scraped data and necessitates a process of removing them.
To solve the following issues, we present SDCNL, and the main points of our work are:
- An application of deep learning-based sentiment analysis for depression versus suicidal ideation classification, which is an important but unexplored clinical and computational challenge
- An entirely new, unsupervised label correction process for text-based data and labels which does not require prior noise distribution information, allowing us to produce large and trustworthy datasets for deep learning
- Strong experimentation on multiple datasets which demonstrates the ability of our approach on a challenging and crucial task
Background
What are the clinical and computational challenges?
Depression remains among the most pressing issues worldwide, especially in developed and emerging countries. According to the World Health Organization (WHO), 322 million people worldwide suffer from depression. Depression can often progress to suicidal ideation and attempted Suicide if left unaddressed.
Given the severity of the issue, diagnosis of depression, and identification of when a depressed individual is at risk of attempting suicide, is an important problem at both the individual and population level. Artificial Intelligence and Machine Learning have emerged as possible ways to quickly, efficiently, and unbiasedly identify struggling users.
Recent research in the field focuses on differentiating mentally unhealthy individuals from clinically healthy ones, and has seen strong success. However, there is little research into finding the line where depression turns into suicidal ideation, which is a more difficult clinical and technological task.
Click here for more information on the problem of depression versus suicide in psychology and mental health: Suicidal: Why We Kill Ourselves, Depression and Suicidality.
Finding data to train these machine learning models is also a huge problem, since data sources in mental health are scarce; most methods rely on EHR data, suicide notes, and more. Online forums where users disclose their conditions and thoughts anonymously and honestly have emerged as popular data sources, and have been frequently used in research due to the high amount of data availability. However, these online datasets also result in lower performance, which can be attributed to the inherent noise in web-scraped labels, justifying the need for a process to correct the noisy labels.
What are Noisy Labels?
While there is a vast amount of data available online, much of it is unusable due to unverifiable labels that may be inaccurate. For our specific problem, labeling data based on subreddit relies on self-reporting, since each user chooses which subreddit they feel best reflects their mental state; thus, they may over or under-report their diagnosis. The concept of labels being corrupted or inaccurate in datasets is referred to as noisy labels. Estimates show that noisy labels can degrade anywhere from 10% to 40% of the dataset, presenting serious challenges for machine learning algorithms.
The issue of noisy labels has been very prevalent in the image-processing domain of machine learning, but has not been addressed in the NLP field. There has been very little work to remove noise from a dataset, and the existing methods rely on extensive prior information, such as noise distribution or having a small amount of trusted data; however, this may not be attainable in real-life scenarios. For example, with our specific task of suicide vs depression classification, we have no prior information about the noise distribution or access to any known, trusted data. This begs the need for a new solution that can identify and remove noise from a dataset in the NLP domain without requiring several prerequisites.
Refer to this review paper for more information about noisy labels and current research addressing the problem.
Our Method

Our method can be broken into 4 main portions: the dataset, word embedding models, noisy label correction, and classification.
Dataset
We develop a primary dataset based on our task of suicide or depression classification. This dataset is web-scraped from Reddit. We collect our data from subreddits using the Python Reddit API. We specifically scrape from two subreddits, r/SuicideWatch and r/Depression. The dataset contains 1,895 total posts. We utilize two fields from the scraped data: the original text of the post as our inputs and the subreddit it belongs to as labels. Posts from r/SuicideWatch are labeled as suicidal, and posts from r/Depression are labeled as depressed. We make this dataset and the web-scraping script available in our code.
We also used two other datasets to verify our label correction and classification methods. The C-SSRS dataset is a dataset that contains 500 Reddit posts from the subreddit r/depression with clinically verified labels. The IMDB dataset is a benchmark NLP dataset with polarized movie reviews, and is often used for binary classification. Additionally, we used a dataset scraped from r/CasualConversation and r/SuicideWatch to compare our method against prior work that focuses on diagnosing depressed versus healthy individuals.
Embedding Models
We first convert the raw text from our posts into numerical representations using word embedding models. We experiment with 3 state-of-the-art transformers: BERT, Sentence-BERT, and the Google Universal Sentence Encoder (GUSE). BERT is a state-of-the-art, bidirectionally trained transformer that achieves high performance on various benchmark NLP tasks. Sentence-BERT is an extension of the original BERT architecture and is retrained and optimized for longer inputs and better performance during clustering. GUSE is a transformer also trained and optimized for greater-than-word length text.
At this point, we have vectorized representations of our text inputs, which our model can work with during our label correction and classification processes.
Label Correction
To address the issue of label noise for our task, we propose an unsupervised label correction method. Our method first uses a dimensionality reduction algorithm to convert the high-dimensional features outputted by the embedding models to lower-dimensional representations. High-dimensional data typically results in subpar performance and poorly separated clusters, a phenomenon known as the "Curse of Dimensionality", meaning we need to reduce it down. From here, we use clustering algorithms to separate these low-dimensional embeddings into two distinct clusters. Specifically, we used Gaussian Mixture Models (GMMs) as our main clustering algorithm, and K-Means Clustering as a baseline. Depending on which cluster each input is in, we can assign a new set of labels based on the clustering predictions. We leverage clustering algorithms because of their unsupervised nature; this is critical because we have no prior knowledge regarding the noise distribution in the labels, requiring a clustering procedure that is independent of the web-scraped labels.
At this point, we have a set of features and two sets of labels: one from the original subreddit, and one from the clustered features. We then use a confidence-based thresholding method to determine which label we should assign to each post. If the clustering algorithm believes a post has above 90% chance of belonging to a certain cluster, then we go with the clustered label. If not, we assign the ground-truth label from the original subreddit. With such a high threshold, we make sure that only the features that definitely have an incorrect label are corrected, preventing any false corrections.
At the end of our label correction method, we have a dataset with features and a new set of labels that have been corrected of their noise. From here, we can use our newly formed dataset for classification.
Classification
After determining the final set of corrected labels, we train our deep learning networks to determine whether the posts display depressive or suicidal sentiment. This ultimately is the final application of our method, and addresses a difficult clinical and technological challenge. Similar to the embedding process, any classifier can be used in place of the ones we tested. However, we aim to prove that deep neural classifiers are effective for our proposed task. The classifiers we experimented with are popular deep learning algorithms, such as DNNs, CNNs, and BiLSTMs.
Results
For our results, we will go over a few of the important outcomes of our label correction and classification tests.
Noisy Label Correction
To evaluate our clustering performance, we present both the accuracy of the clustering algorithm at correcting noisy labels as well as classification performance after label correction. Classification on a clean test set is expected to decrease as training labels become noisier. Therefore, we contend that if after label correction, the classification accuracy of our algorithm increases, the correction method is effective. We evaluate its performance on two datasets, the C-SSRS dataset and the IMDB dataset. We injected artificial noise of up to 40% into the label set and then used our label correction process to remove such noise.

Our noise correction method is able to consistently remove >50% of the injected noise while remaining below a <10% false-correction rate on both datasets, and the performance does not significantly degrade even at higher noise percentages.
Classification
Finally, we show the classification performance of our best performing models. We ran lots of tests with different combinations of classifiers and transformers, and after doing so, we determined the 4 most effective combinations and used them for all the tests. In this section, we will review the performance of those methods before and after label correction, as well as in comparison to the traditional task of suicide vs clinically healthy classification.

On the left, we can see how the label correction process is quite helpful, as the red curves, which are after label correction, are significantly better than the blue ones which are before label correction. This shows both the final ability of our algorithm to perform accurate classification as well as the importance of our label correction process. On the right, when compared to the traditional task of suicidal vs clinically healthy classification, we can see that the classification performance is much higher for the traditional task as opposed to our specific task (prior to label correction). This means that we are addressing a very difficult yet important task, and since no other methods have addressed this specific clinical and psychological problem, our method fills a vital gap.
Final Thoughts
In this article, we have discussed SDCNL, our novel method for classification of depressive sentiment vs suicidal ideation by leveraging deep learning and unsupervised noisy label correction. Our method of label correction using unsupervised clustering effectively removes high-volumes of noise from both benchmark and domain-specific datasets, allowing for the use of large-scale, web-scraped datasets.
The applied setting of our system is to provide professionals with a supplementary tool for individual patient diagnosis, as opposed to solely being a screening method on social media platforms. SDCNL could be used by professional therapists as a "second opinion", friends and family as a preliminary screening for loved ones, or even on social media platforms to identify at-risk users. We contend that our method is applicable in a clinical setting given Reddit’s propensity for honest user disclosure, especially considering the ability of our label correction method to remove substantial amounts of noise from the labels. As opposed to models trained on EHRs or suicide notes, which often include limited amounts of data, leveraging online content allows for the use of data-hungry deep learning models, which we prove can be quite effective.