Easy grammar error detection & correction with Machine Learning

Easy grammar error detection & correction with Machine Learning

Last Updated on 14 July 2021

Machine learning in general and deep learning in particular has boosted Natural Language Processing. A variety of models has allowed to perform machine translation, text summarization and sentiment analysis – to name just a few use cases. Today, we’re adding another one to that list: we’re going to construct a pipeline for grammar error detection & correction with Machine Learning, using Gramformer.

After reading this article, you will…

  • Understand how Transformers can be used for Natural Language Processing.
  • Have built a Gramformer based grammar error detection & correction system with Python.
  • Have built the same system with HuggingFace Transformers instead of the Gramformer repository.

Let’s take a look! 🙂



Transformers for Natural Language Processing

Deep learning based techniques have transformed the field of Machine Learning ever since the breakthrough moment of AI in 2012. While that breakthrough was in the field of Computer Vision, another prominent field where such models have been applied is Natural Language Processing.

Ever since 2017, Transformer based models have been rising in popularity. Before we dive into grammar checking & correction with Gramformer, it is a good idea to provide a brief Transformer background so that everyone understands Gramformer’s context. Click on the link for a more detailed introduction.

Written and spoken text is a sequence of words, and eventually even letters. The combination of letters into words and the combination of words, which is the syntax of e.g. a written text, has underlying semantics, or meaning. This means that when neural networks are to process text, they must be able to handle such meaning. Hence, they must be able to process the text in sequence – or they would fail to capture the meaning. No good would come from a model that mixes all words and letters before processing them, would there?

Traditionally, NLP has worked with recurrent neural networks (such as LSTMs) for handling text. A recurrent neural network is a network where the output of a previous ‘pass’ is passed along to the next, by means of recurrent connections. In other words, the history of what has been processed before during a run (e.g. the words “I was on the way to…” processed before “the supermarket”) is used to predict the next output. In the case of translations, for example, this can be highly useful: translations are sometimes highly dependent on the meaning of what has been produced before.

Precisely this recurrent segment is the bottleneck of recurrent neural networks. It means that every element of the sequence (e.g., every word) has to be processed in sequence. In addition, because LSTMs work with ‘memory’, memory of words processed quite a long time ago (e.g., 20 words ago with long phrases) is faded, possibly hiding semantic dependencies in complex phrases. Using recurrent neural networks and LSTMs in other words was highly ineffective especially with longer sentences.

In 2017, Vaswani et al. produced a completely new architecture for processing language – the Transformer architecture. By applying the attention mechanism in a different way, they showed that attention is all you need – meaning that recurrent segments are no longer necessary. The original Transformer architecture is displayed below and represents N encoder segments and N decoder segments. The encoder segments jointly process text into an intermediary representation, which contains the semantics in a compressed way. This is done by computing multi-head self-attention, a mechanism that essentially allows us to compare the importance of individual words (self-attention) from different angles (multi-head). Once again, please check out the link above if you wish to understand this mechanism in more detail.

The intermediary representations from each encoder segment are then passed into the corresponding decoder segment, as you can see in the image. Where the encoder segment takes a source sequence as its input (e.g. a phrase in Dutch), the decoder takes the corresponding target as its input (e.g. the translation in English). By computing the individual importance of words in the target phrase, and then combining these with the intermediary representation from the source phrase, the model can learn to produce a proper translation.

Beyond translation, which is traditionally performed with such sequence-to-sequence architectures, Transformers have also been applied to text generation (with the GPT-like architectures, using the decoder part) and text interpretation (mostly with BERT-like architectures, using the encoder part).

Let’s now take a look at Gramformer.

The original Transformer architecture, as proposed by Vaswani et al. (2017)

Grammar detection & correction with Gramformer

Gramformer is an open source tool for the detection and correction of grammatical errors in English text:

Gramformer is a library that exposes 3 seperate interfaces to a family of algorithms to detect, highlight and correct grammar errors. To make sure the corrections and highlights recommended are of high quality, it comes with a quality estimator.

Let's pause for a second! 👩‍💻

Blogs at MachineCurve teach Machine Learning for Developers. Sign up to MachineCurve's free Machine Learning update today! You will learn new things and better understand concepts you already know.

We send emails at least every Friday. Welcome!

By signing up, you consent that any information you receive can include services and special offers by email.
GitHub (n.d.)

Grammar detection & correction with Machine Learning – example code

Let’s now take a look at using Gramformer to build a system for grammar detection & correction. Below, you’ll find how to install Gramformer, how to use it for getting corrected text, for getting individual edits, and for getting highlights where errors are detected.

Installing Gramformer

Installing Gramformer is really easy – you can do so using pip (preferably pip3 because of Python 3.x) directly from the Gramformer GitHub repository:

pip3 install -U git+https://github.com/PrithivirajDamodaran/Gramformer.git
Code language: Python (python)

Possible issues when installing Gramformer

  • Issue with lm-scorer
  • Errant not installed
  • En not found https://stackoverflow.com/questions/49964028/spacy-oserror-cant-find-model-en

Getting corrected text

Getting corrected text from Gramformer is quite easy and takes the following steps:

  • Specifying the imports.
  • Fixing the PyTorch seed.
  • Initializing Gramformer.
  • Specifying incorrect phrases.
  • Letting Gramformer give suggestions for phrases including corrections.
  • Printing corrected phrases.

Let’s begin with the imports. We import Gramformer and PyTorch, through torch.

# Imports from gramformer import Gramformer import torch
Code language: Python (python)

Then, we fix the seed. This means that all random number generation is performed with the same initialization vector, and that any deviations can not be related to random number generation.

# Fix seed, also on GPU def fix_seed(value): torch.manual_seed(value) if torch.cuda.is_available(): torch.cuda.manual_seed_all(value) fix_seed(42)
Code language: Python (python)

Then, we initialize Gramformer. We set models to 1, or correction mode, and we instruct it not to use GPU. If you have a dedicated GPU, you can of course set it to True.

# Initialize Gramformer grammar_correction = Gramformer(models = 1, use_gpu=False)
Code language: Python (python)

Let’s then create a list with three gramatically incorrect phrases:

# Incorrect phrases phrases = [ 'How is you doing?', 'We is on the supermarket.', 'Hello you be in school for lecture.' ]
Code language: Python (python)

…after which we can let Gramformer improve them. For each phrase, we let Gramformer perform a correction by suggesting two candidates, and then printing the incorrect phrase with suggested improvements.

# Improve each phrase for phrase in phrases: corrections = grammar_correction.correct(phrase, max_candidates=2) print(f'[Incorrect phrase] {phrase}') for i in range(len(corrections)): print(f'[Suggestion #{i}] {corrections[i]}') print('~'*100)
Code language: Python (python)

As a whole, this yields the following code:

# Imports from gramformer import Gramformer import torch # Fix seed, also on GPU def fix_seed(value): torch.manual_seed(value) if torch.cuda.is_available(): torch.cuda.manual_seed_all(value) fix_seed(42) # Initialize Gramformer grammar_correction = Gramformer(models = 1, use_gpu=False) # Incorrect phrases phrases = [ 'How is you doing?', 'We is on the supermarket.', 'Hello you be in school for lecture.' ] # Improve each phrase for phrase in phrases: corrections = grammar_correction.correct(phrase, max_candidates=2) print(f'[Incorrect phrase] {phrase}') for i in range(len(corrections)): print(f'[Suggestion #{i}] {corrections[i]}') print('~'*100)
Code language: Python (python)

And these are the results when running it:

[Gramformer] Grammar error correct/highlight model loaded.. [Incorrect phrase] How is you doing? [Suggestion #0] ('How are you doing?', -20.39444351196289) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] We is on the supermarket. [Suggestion #0] ("We're in the supermarket.", -32.21493911743164) [Suggestion #1] ('We are at the supermarket.', -32.99837112426758) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] Hello you be in school for lecture. [Suggestion #0] ('Hello, are you in school for the lecture?', -48.61809539794922) [Suggestion #1] ('Hello, you are in school for lecture.', -49.94304275512695) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Code language: PHP (php)

Great! We just built a grammar issue checker & correction tool! 😀

Getting individual edits

Instead of the corrected phrases, we can also print the edits that Gramformer has performed:

# Print edits for each improved phrase for phrase in phrases: corrections = grammar_correction.correct(phrase, max_candidates=2) print(f'[Incorrect phrase] {phrase}') for i in range(len(corrections)): edits = grammar_correction.get_edits(phrase, corrections[i][0]) print(f'[Edits #{i}] {edits}') print('~'*100)
Code language: Python (python)

You can see that is was improved into are for the first phrase; that We is on is turned into We’re in in the second phrase, and so forth.

[Incorrect phrase] How is you doing? [Edits #0] [('VERB:SVA', 'is', 1, 2, 'are', 1, 2)] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] We is on the supermarket. [Edits #0] [('OTHER', 'We is on', 0, 3, "We're in", 0, 2)] [Edits #1] [('VERB:SVA', 'is', 1, 2, 'are', 1, 2), ('PREP', 'on', 2, 3, 'at', 2, 3)] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] Hello you be in school for lecture. [Edits #0] [('OTHER', 'Hello', 0, 1, 'Hello,', 0, 1), ('VERB', '', 1, 1, 'are', 1, 2), ('VERB', 'be', 2, 3, '', 3, 3), ('DET', '', 6, 6, 'the', 6, 7), ('NOUN', 'lecture.', 6, 7, 'lecture?', 7, 8)] [Edits #1] [('OTHER', 'Hello', 0, 1, 'Hello,', 0, 1), ('MORPH', 'be', 2, 3, 'are', 2, 3)] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Code language: CSS (css)

Getting highlights

Simply changing get_edits into highlight will yield the original phrase where the errors are marked:

# Print highlights for each improved phrase for phrase in phrases: corrections = grammar_correction.correct(phrase, max_candidates=2) print(f'[Incorrect phrase] {phrase}') for i in range(len(corrections)): highlights = grammar_correction.highlight(phrase, corrections[i][0]) print(f'[Highlights #{i}] {highlights}') print('~'*100)
Code language: Python (python)

In other words:

Never miss new Machine Learning articles ✅

Blogs at MachineCurve teach Machine Learning for Developers. Sign up to MachineCurve's free Machine Learning update today! You will learn new things and better understand concepts you already know.

We send emails at least every Friday. Welcome!

By signing up, you consent that any information you receive can include services and special offers by email.
[Gramformer] Grammar error correct/highlight model loaded.. [Incorrect phrase] How is you doing? [Highlights #0] How <c type='VERB:SVA' edit='are'>is</c> you doing? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] We is on the supermarket. [Highlights #0] <c type='OTHER' edit='We're in'>We is on</c> the supermarket. [Highlights #1] We <c type='VERB:SVA' edit='are'>is</c> <c type='PREP' edit='at'>on</c> the supermarket. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Incorrect phrase] Hello you be in school for lecture. [Highlights #0] <a type='VERB' edit='<c type='OTHER' edit='Hello,'>Hello</c> are'><c type='OTHER' edit='Hello,'>Hello</c></a> you <d type='VERB' edit=''>be</d> in school <a type='DET' edit='for the'>for</a> <c type='NOUN' edit='lecture?'>lecture.</c> [Highlights #1] <c type='OTHER' edit='Hello,'>Hello</c> you <c type='MORPH' edit='are'>be</c> in school for lecture. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Code language: HTML, XML (xml)

Using Gramformer with HuggingFace Transformers

According to the setup.py installation instructions, Gramformer is built on top of HuggingFace Transformers. This means that you can also construct Gramformer with HuggingFace Transformers, meaning that you don’t need to install the Gramformer repository with pip. Here’s an example that illustrates how you can use the AutoTokenizer and AutoModelForSeq2SeqLM with the pretrained Gramformer tokenizer/model for grammar checking:

# Imports from transformers import AutoTokenizer, AutoModelForSeq2SeqLM # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained("prithivida/grammar_error_correcter_v1") # Load the model model = AutoModelForSeq2SeqLM.from_pretrained("prithivida/grammar_error_correcter_v1") # Incorrect phrases phrases = [ 'How is you doing?', 'We is on the supermarket.', 'Hello you be in school for lecture.' ] # Tokenize text tokenized_phrases = tokenizer(phrases, return_tensors='pt', padding=True) # Perform corrections and decode the output corrections = model.generate(**tokenized_phrases) corrections = tokenizer.batch_decode(corrections, skip_special_tokens=True) # Print correction for i in range(len(corrections)): original, correction = phrases[i], corrections[i] print(f'[Phrase] {original}') print(f'[Suggested phrase] {correction}') print('~'*100)
Code language: Python (python)

…results:

[Phrase] How is you doing? [Suggested phrase] How are you doing? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Phrase] We is on the supermarket. [Suggested phrase] We are at the supermarket. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Phrase] Hello you be in school for lecture. [Suggested phrase] Hello you are in school for lecture. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Code language: CSS (css)

Summary

In this article, you have…

  • Found how Transformers can be used for Natural Language Processing.
  • Built a Gramformer based grammar error detection & correction system with Python.
  • Built the same system with HuggingFace Transformers instead of the Gramformer repository.

I hope that it was useful and that you have learned a lot. Thank you for reading MachineCurve and happy engineering! 😎


Sources

GitHub. (n.d.). PrithivirajDamodaran/Gramformerhttps://github.com/PrithivirajDamodaran/Gramformer

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you needAdvances in neural information processing systems30, 5998-6008.

GitHub. (2020). Python 3.8 support? · Issue #10 · simonepri/LM-scorerhttps://github.com/simonepri/lm-scorer/issues/10

Do you want to start learning ML from a developer perspective? 👩‍💻

Blogs at MachineCurve teach Machine Learning for Developers. Sign up to learn new things and better understand concepts you already know. We send emails every Friday.

By signing up, you consent that any information you receive can include services and special offers by email.

Leave a Reply

Your email address will not be published. Required fields are marked *