How to understand a language modeling objective, intuitively?

Ask Questions Forum: ask Machine Learning Questions to our readersCategory: Machine LearningHow to understand a language modeling objective, intuitively?
Chris Staff asked 11 months ago
1 Answers
Best Answer
Chris Staff answered 9 months ago

From What is the Longformer Transformer and how does it work?
 

Recall that Autoregressive Language Modeling (LM) involves “estimating the probability distribution of an existing token/character given its previous token/characters in an input sequence”. In other words, it involves predicting the next token given previous tokens using some maximum likelihood.

Your Answer

4 + 17 =