BERT (language model)

BERT, short for Bidirectional Encoder Representations from Transformers, is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning, and it uses the encoder-only transformer architecture. Training combines masked token prediction with next sentence prediction, which lets the model learn contextual, latent representations of each token in its surrounding text, in the manner of ELMo and GPT-2. BERT dramatically improved the state of the art for large language models and found applications in many natural language processing tasks, such as coreference resolution and polysemy resolution. It also gave rise to the study of BERTology, which tries to interpret what the model has learned. The original release came in two English-language sizes, BERT-base with 110 million parameters and BERT-large with 340 million parameters, both trained on the Toronto BookCorpus and English Wikipedia, with the weights published on GitHub. In March 2020 twenty-four smaller models followed, the smallest having four million parameters.