sourced story
11 October 2018Primary source · 2 sourcesWell documented

BERT makes language models read in both directions

Google's model masks random words and learns to guess them from context on both sides

On the timeline · around 11 October 2018 · Transformers and the Generative AI WaveThe Statistical and Machine Learning TurnTransformers and the Generative AI WaveBERT makes language models read in both directions2015201820192020

Quick facts

Lead author
Jacob Devlin
Institution
Google AI Language
Key innovation
Bidirectional masked-language-model pretraining

What happened

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova at Google AI Language published 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.' Earlier Transformer-based language models, including OpenAI's original GPT, read text strictly left to right, so each word could only be informed by the words before it. BERT instead trained using a 'masked language model' objective, randomly hiding some words in a sentence and training the model to predict them using context from both the left and the right simultaneously, giving it a genuinely bidirectional understanding of a sentence. The pretrained model could then be fine-tuned with just one additional output layer for a specific task, and it set new state-of-the-art results across eleven different natural language processing benchmarks, pushing the GLUE benchmark score up by 7.7 percentage points over the prior best. Google open-sourced the model and pretrained weights the following month.

Why it matters

BERT showed that bidirectional context, understanding a word using everything around it rather than only what precedes it, produced substantially better language representations, and its open-sourced weights let researchers everywhere fine-tune a state-of-the-art model without training one from scratch.

How we know

The original arXiv paper documents the masked-language-model method and the eleven-benchmark results directly; Google's own open-sourcing announcement corroborates the bidirectional design and its practical implications for other researchers.

Sources

See something wrong? . Corrections with a source get fixed fastest.

Part of a timelineArtificial Intelligence30 events · From a wartime theory of neurons to machines that write, paint, and fold proteinsView all →
BERT makes language models read in both directions · Artificial Intelligence · SourcedStory