ÉtudeScience

Computer Science & AI · 2017

Attention Is All You Need

Ashish Vaswani, et al. (Google Brain / Google Research)

Google Brain

Advances in Neural Information Processing Systems (NeurIPS), vol. 30 ·

Cited by 180,000+Open access
View original paper

Opens the version of record via arXiv

The paper that introduced the Transformer, an architecture built entirely on attention mechanisms with no recurrence or convolution. It became the foundation of modern large language models such as GPT and BERT.

The architecture behind the large-language-model revolution.

  • Introduced the Transformer, relying solely on self-attention to model sequences.
  • Removed recurrence, enabling far more parallel training and longer-range dependencies.
  • Set new state-of-the-art results on machine translation at a fraction of the training cost.

An encoder–decoder neural network using multi-head self-attention and position encodings, evaluated on machine-translation benchmarks against recurrent and convolutional baselines.

Keywords

Computer Science & AI

Highly Accurate Protein Structure Prediction with AlphaFold

Jumper et al. · 2021 · Nature

AlphaFold solved a 50-year grand challenge in biology: predicting a protein's three-dimensional shape from its amino-acid sequence. Its accuracy approached that of experimental methods, transforming structural biology.

Cited by 37,000+Open access

Computer Science & AI

Deep Learning

LeCun, Bengio & Hinton · 2015 · Nature

A landmark review by three pioneers of the field, synthesising how deep neural networks learn layered representations of data. It explains why depth works and surveys the advances that made deep learning dominant across vision, speech, and language.

Cited by 110,000+

Computer Science & AI

Generative Adversarial Nets

Goodfellow et al. · 2014 · Advances in Neural Information Processing Systems (NeurIPS)

Goodfellow and colleagues introduced generative adversarial networks (GANs), a way to train generative models by pitting two networks against each other — one creating fake data, the other trying to detect it — until the fakes are indistinguishable from real data.

Cited by 70,000+Open access

Étude Science indexes and summarises this work; it is not the publisher. The summary above is written by Étude. For the definitive text, figures, and data, please consult the original publication via the link above. Vaswani et al. (2017) hold the rights to the original work.