BERT (Bidirectional Encoder Representations from Transformers) revolutionized NLP research in 2018 by showing that simple Masked Language Modeling is an extremely powerful pretraining signal.
The Core Problem: ELMo vs. BERT
Before BERT, there were two paradigms:
ELMo (Bidirectional):
- Train two separate LSTMs
- One forward, one backward
- Combine both
- Result: Context-aware embeddings
But: Slow, requires two separate models
GPT (Unidirectional):
- One Transformer, reads left-to-right
- Faster
- But: Can't see information to the right
BERT insight: We can train a Transformer that's bidirectional WITHOUT making it unidirectional.
Masked Language Modeling (MLM)
Core idea is simple:
Original sentence:
"The dog jumped over the fence"
Masking (15% of tokens):
"The [MASK] jumped over the [MASK]"
Task: Predict masked tokens
Expected: dog, fence
Different from GPT:
- GPT: "Predict next token"
- BERT: "Predict random token in the middle"
Why does this work?
Model must use context from both sides:
GPT at "The dog ...":
- Can only look right
- dog is unknown
- Must guess
BERT at "The [MASK] jumped ...":
- Can see "The" on left
- Can see "jumped" on right
- Context: "something that jumps" → "dog"
- Higher information density
Next Sentence Prediction (NSP)
BERT also trained on:
Two sentences, random:
Sent A: "The cat sits on the mat"
Sent B (50%): "It sleeps there" (follows A)
Sent B (50%): "Fish swim in water" (random)
Task: Predict if B follows A
Adds sentence-level understanding.
Note: Later research showed NSP probably isn't necessary. But seemed logical at the time.
BERT Architecture
BERT-Base:
- 12 Transformer blocks
- 768 hidden dimension
- 12 attention heads
- 110 million parameters
- Vocab: 30,000 tokens
BERT-Large:
- 24 Transformer blocks
- 1024 hidden dimension
- 16 attention heads
- 340 million parameters
Training
Data:
- BookCorpus (800M words)
- Wikipedia (2.5B words)
- Total: 3.3 trillion tokens
Hardware:
- 64 TPU v3 chips
- Training time: 4 days
Hyperparameters:
- Learning rate: 0.0001
- Batch size: 256
- Optimizer: Adam
The Fine-Tuning Paradigm
BERT's greatest contribution: Show that pretraining + fine-tuning simply works.
Phase 1: Pretraining
- Train on huge unlabeled dataset
- Expensive (but one-time)
Phase 2: Fine-tuning
- Take BERT
- Add simple classification layer
- Train on small labeled dataset
- Needs only hours
Concrete Examples
Text Classification (Sentiment):
Input: "This product is great!"
[CLS] token → Classification layer → Positive
Named Entity Recognition:
Input: "Donald Trump visited New York"
Output: [Person] [Person] [Location]
Question Answering:
Question: "What's France's capital?"
Context: "France has Paris as capital"
Output: "Paris" (character position in context)
Impact and Adoption
Benchmark improvements after BERT:
GLUE score:
Before BERT: 83.7%
BERT-Base: 84.6%
BERT-Large: 87.4%
SQuAD (QA):
Before: 83.1% F1
BERT: 93.2% F1
BERT was so successful that practically all subsequent NLP models used the same paradigm.
Evolution: RoBERTa, ALBERT, DeBERTa
After BERT came several improvements:
RoBERTa (Facebook, 2019)
Improvements over BERT:
1. Trained longer (1M steps, not 100K)
2. Larger vocabulary (50K tokens)
3. No NSP (not necessary)
4. Better hyperparameter tuning
5. Byte-level BPE tokenizer
Result: +2-3% on benchmarks
ALBERT (Google, 2019)
Focus: Reduce size
Techniques:
1. Parameter sharing between layers
2. Factorized embeddings
3. Sentence order prediction instead of NSP
Result: 1/10 parameters of BERT, similar performance
DeBERTa (Microsoft, 2020)
Focus: Better attention
Innovations:
1. Disentangled attention (separate content vs. position)
2. Enhanced mask decoder
Result: +2-4% over BERT on benchmarks
Why BERT is Legacy
BERT wasn't the largest or fastest model, but:
- Proved a simple pattern: Large pretraining + simple fine-tuning
- Scaled to many languages: multilingual BERT
- Enabled fine-tuning ecosystem: Hundreds of BERT-based models
- Was first with huge success: Timing matters
BERT Timeline
2018: BERT released
2019: RoBERTa, ALBERT, ELECTRA
2020: DeBERTa, ERNIE
2021: Efficient models: DistilBERT, TinyBERT
2022-2024: Decoder models dominate (GPT, LLaMA)