Skip to content
mlmentorship

BERT and masked language modeling

Train a transformer to fill in randomly masked tokens. The result is a bidirectional encoder that broke a dozen NLP benchmarks at once and defined the pretrain-then-finetune era.

Published · 6 min read ·Specialist ·Intermediate

Visual quick review

Visual first · depth when needed

Distinguish the 15 percent target-selection step from the conditional 80/10/10 input-corruption step, and identify exactly which positions receive direct masked-language-modeling loss.

Preparing the visual…

Summary

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al., 2019) is a transformer encoder pretrained with masked language modeling: replace 15 percent of input tokens with a special [MASK] token, train the model to predict them. The pretrained encoder is then fine-tuned for downstream tasks.

Pre-BERT, NLP pipelines were task-specific: parse trees for parsing, sequence-to-sequence for translation, hand-crafted features for classification. BERT showed that one bidirectional pretrained encoder, fine-tuned per task, beat the entire task-specific stack on 11 benchmarks at once.

The BERT recipe (pretrain on raw text, fine-tune per task) defined NLP from 2018 to roughly 2022. Decoder-only LLMs (GPT family) eventually dominated for generative work, but BERT-style encoders are still the right answer for classification, retrieval, and embedding tasks. Most production embedding models (Sentence-BERT, modern retrieval encoders) are BERT descendants.

The pretraining task

Masked Language Modeling (MLM)

Pick 15 percent of token positions. Of those:

  • 80 percent are replaced with [MASK].
  • 10 percent are replaced with a random token.
  • 10 percent are kept as the original token.

Train the model to predict the original token at each picked position, using cross-entropy. The loss is evaluated only at the picked positions. The remaining 85 percent have no direct prediction loss, but they still supply context and can affect the selected predictions.

The 10/10 random/keep split exists because at fine-tuning time there are no [MASK] tokens. The model needs to handle every input position consistently.

Learning objective

Which tokens become MLM targets, and what input does BERT see at those positions?

Fifteen percent of positions become prediction targets All input positions provide bidirectional context. A random 15 percent are selected as MLM targets and receive a direct cross-entropy loss. The other 85 percent receive no direct prediction loss, although they can influence predictions as context. 1 · CHOOSE TARGET POSITIONS 100% of input positions all supply left + right context randomly select selected 15% predict original token direct cross-entropy loss other 85% context for targets no direct MLM loss SELECTION SETS WHERE LOSS IS MEASURED
Every selected position follows one of three input-corruption paths Given the selected original token dinner, 80 percent of the time BERT sees MASK, 10 percent it sees a random token such as violin, and 10 percent it sees dinner unchanged. On every path the training target remains dinner and the selected position receives loss. 2 · CORRUPT EACH SELECTED POSITION original target dinner 80% · [MASK] 10% · violin 10% · dinner predict dinner All 3 paths keep the same target and receive loss. 80 / 10 / 10 IS CONDITIONAL ON SELECTION
Read it this way: first choose the 15 percent of positions where BERT must recover the original token. Only then choose what input replaces each selected token: `[MASK]`, a random token, or the unchanged token. All three paths predict the same original target; unselected positions provide context without their own MLM loss.

Why bidirectional matters

A causal LM (GPT-style) only attends to previous tokens. A masked LM has access to context on both sides. For tasks like classification, NER, or extractive QA where the full input is available, bidirectional context is strictly more informative.

Next Sentence Prediction (NSP)

The original BERT also predicted whether two sentences appeared consecutively in the corpus. Subsequent work (RoBERTa) showed NSP adds little; modern variants drop it.

Architecture

Standard transformer encoder. Inputs:

  • Token embeddings (WordPiece subwords).
  • Position embeddings (learned).
  • Segment embeddings (which of two sentences the token belongs to).

Special tokens:

  • [CLS] at position 0. Its final-layer hidden state is used as the sequence representation for classification.
  • [SEP] between sentences and at the end.

BERT-base: 12 layers, 768 hidden dim, 12 heads, 110M parameters. BERT-large: 24 layers, 1024 hidden, 16 heads, 340M parameters.

Fine-tuning

Add a small head on top of the pretrained encoder, train end-to-end on the downstream task:

TaskHead
Single-sequence classificationLinear on [CLS]
Sentence-pair classification (NLI)Linear on [CLS], both sentences in input
Token classification (NER, POS)Linear on every token’s final hidden state
Extractive QATwo linears predicting span start and end positions

Typical fine-tune: 2 to 5 epochs, learning rate , small batch.

Variants

  • RoBERTa (Liu et al., 2019). More data, longer training, no NSP, dynamic masking. The “BERT done right” reference.
  • ALBERT (Lan et al., 2019). Parameter sharing across layers, factorized embeddings.
  • DeBERTa (He et al., 2021). Disentangled position and content attention.
  • Sentence-BERT (Reimers & Gurevych, 2019). BERT fine-tuned with siamese training to produce sentence embeddings useful with cosine similarity.

When to use BERT in 2026

  • Classification, NER, extractive QA: still competitive and much smaller than an LLM.
  • Embeddings for retrieval: the modern stack (E5, BGE, GTE) is BERT-family.
  • Anywhere bidirectional context helps and you do not need free-form generation.

When to skip: anything generative. Use a decoder-only LLM.

Common pitfalls

  • Forgetting that fine-tuning is full backprop through the encoder. Freeze the encoder only if you cannot afford otherwise; full fine-tuning is the strong baseline.
  • Using the [CLS] representation directly for sentence similarity. It was not pretrained for that. Use Sentence-BERT or one of its descendants instead.
  • Treating BERT as a generative model. It cannot generate left-to-right text; the masking objective is local.

Found an error or stale claim? Report a correction or send a private note.