Grammatical Error Correction with Machine Learning - Overview and Implementation

1. Introduction

Natural language processing (NLP) pipelines rely on Machine Learning models that consume, analyze and / or transform textual data for various purposes. For example, Google Translate receives the incoming text in one language and returns an outgoing text in the target language (this task is called Machine Translation). Sentiment analysis algorithms, receive textual data and determine whether the text is positive, negative or neutral. Text summarization models, receive textual inputs and summarize them into smaller textual outputs. There are many factors that can impact the performance / output quality of such models and one of them is the quality of the incoming text. Specifically, noise, in the form of errorful text, can adversely impact the outcome quality of neural machine translation models (Belinkov and Bisk, 2018). Therefore, there have been efforts focused on improving the grammatical correctness of the incoming textual data across NLP pipelines, before such textual data reach the downstream tasks of machine translation, sentiment analysis, text summarization, etc.

Grammatical error correction models generally use two approaches:

Sequence-to-sequence (seq2seq) text generation, which can be thought as a translation engine that translates from a given language to the same language, while correcting the grammatical errors (e.g. Yuan and Briscoe, 2014)
Sequence tagging, where incoming text is tokenized, tagged and then mapped back to corrected tokens (e.g. Malmi, et al., 2019)

While seq2seq neural machine translation approach has been documented to achieve state-of-the-art performance (e.g. Vaswani et al., 2017), but it still suffers from certain shortcomings, such as: (1) Inference and generation of outputs take a long time, (2) Training requires large amounts of data, and (3) Neural architecture of the model makes interpretation of the results challenging, compared to non-neural architectures ([Omelianchuk, et al., 2020](https://aclanthology.org/2020.bea-1.16/)). In order to overcome these shortcomings, the approach that we will be talking about and then implementing in this post is a sequence tagger that uses a Transformer encoder. Omelianchuk, et al., 2020’s work is pre-trained on synthetic data. Then the pre-trained models are fine-tuned in two stages. One stage purely includes errorful corpora and then the second fine-tuning stage includes a combination of errorful and error-free data. The resulting work is up to ten times as fast as a Transformer seq2seq system and is publicly available on GitHub. This approach improves the inference time concern of seq2seq models and can achieve a higher level of customization given a smaller training data, since it is based on a pre-trained model but still leaves interpretability and explainability as an improvement opportunity for future work.

In the next section, we will use this library to implement an approach to correct grammatical errors in a given sentence. Then we will create a visual user interface to demo the results.

2. Grammatical Error Correction – Implementation

I will break this section down into three steps:

Prepare the Requirements: This step includes cloning the repository, downloading the pre-trained model and installing the requirements needed to implement the grammatical error correction model. I use the command-line interface (CLI) for these steps.
Model Implementation: Implement and test the grammatical error correction model. I implement these steps in a Jupyter notebook.
User Interface: Create a user interface to enhance user experience

2.1. Prepare the Requirements

The first step to prepare the requirements is to clone the publicly-available repository into our local system. In other words, we will create a copy of the library from GitHub into our computer, using the following command:

git clone https://github.com/grammarly/gector.git

There are three pre-trained models available. For this part of the exercise, we are going to rely on the one using RoBERTa as the pre-trained encoder, which has the highest overall score among the existing models. Let’s go ahead and download the pre-trained model, using the following command:

wget https://grammarly-NLP-data-public.s3.amazonaws.com/gector/roberta_1_gectorv2.th

Now that we have the model downloaded to our local, I am going to move it to the "gector" directory, which is in the directory that we cloned from GitHub, using the following command:

mv roberta_1_gectorv2.th ./gector/gector

Next, we will go to the appropriate directory to start running the model, using the following command:

cd ./gector

This package relies on other libraries to execute so we are going to install these requirements, with the following command:

pip install -r requirements.txt

Now we have all the files in the right places to start creating the grammatical error correction model in the next step.

2.2. Implement the Model

Now that we have all the directories and files needed for this model, we are going to start using the library. We will take the following steps:

Import the necessary packages
Create an instance of the model
Test the model on a sentence with grammatical errors to see the output. For this purpose, we will use the following sentence: "she are looking at sky". What do you expect the corrected sentence to be? Write that down and compare it to the outcome!

# Import libraries
from gector.gec_model import GecBERTModel

# Create an instance of the model
model = GecBERTModel(vocab_path = "./data/output_vocabulary", model_paths = ["./gector/roberta_1_gectorv2.th"])

# Add the sentence with grammatical errors
sent = 'she are looking at sky'

# Create an empty list to store the 
batch = []
batch.append(sent.split())
final_batch, total_updates = model.handle_batch(batch)
updated_sent = " ".join(final_batch[0])
print(f"Original Sentence: {sent}n")
print(f"Updated Sentence: {updated_sent}")

Results:

Updated sentence is quite amazing! Let’s look at the changes:

Capitalized "she" to "She" at the beginning of the sentence
Changed "are" to "is" to have subject-verb agreement for "she" and "is"
Added "the" before "sky"
Added a period to the end of the sentence

These all are good changes and they are exactly what I would have done myself, if I were to correct the sentence, but….what if we had a more complicated sentence? Let’s mix tenses and see how the model performs.

# Add the sentence with grammatical errors
sent = 'she looks at sky yesterday whil brushed her hair'

# Create an empty list to store the 
batch = []
batch.append(sent.split())
final_batch, total_updates = model.handle_batch(batch)
updated_sent = " ".join(final_batch[0])
print(f"Original Sentence: {sent}n")
print(f"Updated Sentence: {updated_sent}")

Results:

This is also very interesting. Let’s summarize the changes:

Capitalized "she" to "She" at the beginning of the sentence
Changed "looks" to "looked", which is now in agreement with "yesterday"
Added "the" before "sky"
Added the missing letter to "while"
Changed "brushed" to "brushing", which is the expected form after "while"

Notice that the model decided the intended verb tense is the past. Another approach could have been to decide that the intended verb tense is present and change "yesterday" to "today" but based on the trained data, the model decided to go with the past tense.

Now let’s look at one more example and see if we can push the boundaries of the model and confuse it with tenses:

# Add the sentence with grammatical errors
sent = 'she was looking at sky later today whil brushed her hair'

# Create an empty list to store the 
batch = []
batch.append(sent.split())
final_batch, total_updates = model.handle_batch(batch)
updated_sent = " ".join(final_batch[0])
print(f"Original Sentence: {sent}n")
print(f"Updated Sentence: {updated_sent}")

Results:

Finally we have found an edge case where the model does not recognize the correct verb tense. The updated sentence is about "later today", which implies the future tense, while the model generates the sentence in the past tense. So why is this more challenging to the model than before? The answer is that "later today" is implying time in two words, which requires a deeper level of contextual awareness from the model. Note that without the word "later", we would have had a completely acceptable sentence as follows:

In this context, "today" could be referring to earlier today (i.e. in the past), which would make the grammatical correction completely acceptable. But in the original example, "later today" is not recognized by the model as an indicated of the future tense. As a general note, it is a good practice to test out these models on various use cases to be aware of such limitations.

2.3. User Interface

Now that we have gone through a few examples, we will make two updates to improve user experience through a user interface:

Create a function that accepts a sentence and returns the updated (i.e. grammatically-corrected) sentence
Add a visual interface for ease of use

# Define a function to correct grammatical errors of a given sentence
def correct_grammar(sent):
    batch = []
    batch.append(sent.split())
    final_batch, total_updates = model.handle_batch(batch)
    updated_sent = " ".join(final_batch[0])
    return updated_sent

Let’s test the function on one of our sentences and make sure it works as intended.

sent = 'she looks at sky yesterday whil brushed her hair'

print(f"Original Sentence: {sent}n")
print(f"Updated Sentence: {correct_grammar(sent = sent)}")

Results:

The function performs as expected. Next we will add a visual user interface to improve user experience. For this purpose, we are going to use Gradio, which is an open-source Python library to create demos and web applications, as we will see below.

Hint: If you do not have this installed, you can install it with the following command:

pip install gradio

With Gradio installed, let’s continue with importing and creating the user interface as follows:

# Import Gradio
import gradio as gr

# Create an instance of the Interface class
demo = gr.Interface(fn = correct_grammar, inputs = gr.Textbox(lines = 1, placeholder = 'Add your sentence here!'), outputs = 'text')

# Launch the demo
demo.launch()

Results:

UI of the Grammatical Error Correction Model (Using Gradio)

Now that we have the demo interface, let’s test our sentence again and see how it works! We simply type the sentence in the box on the left side and press "Submit". Then the results will show up in the box in the right hand side as follows:

Grammatical Error Correction Model Results (Using Gradio)

The demo is working as expected. Give it a shot and test other sentences to see how it works!

3. Conclusion

In this post we looked at the benefits of adding a grammatical error correction model to existing NLP pipelines, such as neural machine translation. We then reviewed two categories of approaches currently used in the literature to tackle this task. And finally we used a publicly-available library to implement a pre-trained grammatical error correction model, tested it on a few errorful sentences, found where the model works well and also faced some limitations and finally added a visual interface to improve user experience.