In a Masked Language Modeling task, language models don’t have access to the full input – but rather to a masked input, where some (10-20 percent) of the input tokens are masked. This simply means replacing the tokens (or some token spans) with a special token representing <mask>. The goal for the MLM task becomes reconstructing the original sequence, i.e. to reveal what is hidden under the mask. This adds complexity on top of regular language modeling tasks, and some works argue that it can help boost performance.
 

 
Source:
 
MachineCurve. (2021, March 2). Easy masked language modeling with machine learning and HuggingFace transformers. https://www.machinecurve.com/index.php/2021/03/02/easy-masked-language-modeling-with-machine-learning-and-huggingface-transformers/