'Attention Is All You Need' introduces the Transformer
Google researchers drop recurrence entirely and let every word attend to every other word at once
Quick facts
- Lead author
- Ashish Vaswani
- Institution
- Google Brain / Google Research
- Key innovation
- Self-attention, no recurrence or convolution
What happened
Ashish Vaswani and seven co-authors at Google Brain and Google Research published 'Attention Is All You Need,' introducing the Transformer architecture. Earlier sequence models processed text word by word in order, using recurrent connections that made long-range relationships hard to learn and training difficult to parallelize. The Transformer replaced recurrence entirely with a self-attention mechanism, which lets the model weigh every word in a sentence against every other word simultaneously to build a representation informed by the whole context at once, rather than only what came before. Because self-attention can be computed for all words in parallel rather than one step at a time, Transformers trained dramatically faster on modern GPU hardware. The model achieved a new state-of-the-art BLEU score of 41.8 on an English-to-French translation benchmark after training for just 3.5 days on eight GPUs, far less compute than prior best results required.
Why it matters
The Transformer's parallelizable attention mechanism became the shared architectural foundation for nearly every major language model that followed, including BERT, GPT, and the large language models behind ChatGPT, because it scaled to much larger datasets and models than recurrent architectures could practically train.
How we know
The original arXiv paper documents the architecture and translation benchmark results directly; Google's own research blog post on the paper independently explains the same self-attention mechanism using a worked example of resolving word ambiguity.
Sources
- Ashish Vaswani, Noam Shazeer, Niki Parmar, et al.. Attention Is All You Need · Primary source (author-declared)arxiv.org · Cited as a "primary" source (no stronger domain match). · Link is live and its text matches the event's key terms (Jul 2026)
- Google Research Blog. Transformer: A Novel Neural Network Architecture for Language Understanding · General sourceai.googleblog.com · Cited as a "reference" source (no stronger domain match). · Link is live and its text matches the event's key terms (Jul 2026)
See something wrong? . Corrections with a source get fixed fastest.
Part of a timelineArtificial Intelligence30 events · From a wartime theory of neurons to machines that write, paint, and fold proteinsView all →