Skip to content
mlmentorship

Design a content moderation system

Moderation is a multi-policy classification problem at scale, with appeals, human review, and adversarial users. The senior answer separates policy from model and treats human review as part of the system.

Published · 7 min read ·Core ·Advanced

ML system design · active recall

Practice before you read

40 minutes. Scope first; then cover data, baseline, model, evaluation, serving, monitoring, and iteration.

Back to my workbook How practice works

ML system design · closed-book attempt

Design a content moderation system

Scope first; then cover data, baseline, model, evaluation, serving, monitoring, and iteration.

Problem

Context. A hypothetical platform must assess incoming content under several policies, with different error costs across categories and languages. Automated scores are imperfect, and human review has limited capacity. Task. Define policy labels, decision thresholds, automated actions, review and appeal paths, and a measurement and rollout plan. Explain how policy changes and adversarial behavior affect operation. Constraints. Separate policy authority from model prediction. Specify false-positive and false-negative costs rather than optimizing accuracy alone. Protect user data, audit decisions, and provide a response when uncertainty exceeds the system's authority.

40:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.

Asked in: trust-and-safety, social-platform, and LLM-deployment interviews.

Design a content-moderation system that applies explicit policies while accounting for uncertainty, review capacity, and user appeals.

Problem

Context. A hypothetical platform must assess incoming content under several policies, with different error costs across categories and languages. Automated scores are imperfect, and human review has limited capacity.

Task. Define policy labels, decision thresholds, automated actions, review and appeal paths, and a measurement and rollout plan. Explain how policy changes and adversarial behavior affect operation.

Constraints. Separate policy authority from model prediction. Specify false-positive and false-negative costs rather than optimizing accuracy alone. Protect user data, audit decisions, and provide a response when uncertainty exceeds the system’s authority.

What makes moderation hard

  1. Policy is fuzzy and contested. “Hate speech” has no universal definition; the platform’s policy is a specification.
  2. Users adapt. What’s flagged today is rephrased tomorrow.
  3. High-stakes both ways. Over-moderation suppresses legitimate speech; under-moderation harms users.
  4. Scale. Billions of items; can’t human-review everything.
  5. Multi-policy. Hate speech, sexual content, violence, spam, fraud, copyright are different tasks with different policies and costs.

What an L5 answer sounds like

“Architecture, in layers:

Layer 1: triage by score. A multi-task classifier (or one model per policy) outputs scores for each policy violation. Most content is low-risk and gets allowed. High-risk content is flagged.

Layer 2: action determination. Per policy, decide what to do based on the score:

  • Below low threshold: allow.
  • Between low and high thresholds: route to human review.
  • Above high threshold: auto-action (remove, reduce reach, age-gate, depending on policy).

Layer 3: human review. Trained moderators review the routed content. Their decisions feed back as labels for retraining.

Layer 4: appeals. Users can appeal automated actions; appeals route to senior reviewers; final decisions feed back as labels.

Eval: per-policy precision and recall at multiple thresholds. Calibrate against trained moderator decisions on a held-out set. A/B-test policy threshold changes with explicit business and harm-floor metrics.”

This is L5. Layered architecture with human-in-the-loop and feedback.

What an L6 answer adds

“…practical things:

Policy and model are separate concerns. The platform decides policy; ML implements it. Conflating the two creates ML teams that look like they’re making policy decisions (which they shouldn’t be unilaterally). Document the policy in plain language; build the model to that spec; track ‘model agreement with policy spec’ as a metric, not ‘model agreement with current model behavior.’

Per-policy thresholds, not a single ‘bad’ threshold. Different policies have different cost structures. Hate speech: high cost of false negative, willing to tolerate more false positives. Spam: high volume, lower cost per error, optimize differently.

Adversarial robustness. Users will test the system, find phrasings that bypass it, and share. Augment training data with known evasion patterns; track ‘evasion rate’ (flagged content that mutates to unflagged versions); maintain a fast-update path for new patterns.

Audit relevant slices. Measure probability quality, false positives, false negatives, and policy consistency across languages and other appropriate slices. An accuracy gap is not itself a calibration diagnosis. Group-specific calibration or thresholds require a justified policy and an assessment of legal, privacy, fairness, and data-support constraints.

Route review by risk and uncertainty. Near-threshold cases can be useful, but a confidently wrong extreme can also be consequential. Reserve some representative and risk-targeted audits outside the borderline queue; measure reviewer agreement and capacity.

Counterfactual eval for false negatives. False positives are visible (users complain). False negatives are invisible by definition (the bad content stayed up). Track via periodic random audits and via reports from users / partners.

Appeals are valuable but selected evidence. Appealers differ from users who cannot or do not appeal, and reviewers can disagree. Version the policy and adjudication, audit label quality, and keep representative sampling separate. As in the fraud-design lesson, observed outcomes depend on the policy; appeal accuracy alone cannot estimate population error rates.

Explainability is a regulatory and trust issue. When auto-actioning content, surface a reason (‘flagged for hate speech against [protected category]’); helps user understanding and reduces disputes.”

Score 0.62 means review only under the declared policy P7

A score gets meaning from the versioned policyScore 0.62 lies inside this illustrative P7 review region. It is not intrinsically a permission to publish or remove content.Policy P7 maps a score coordinate toan action regionallowreviewblock0.620.62 lies inside the declared 0.4–0.8review region.
Derivation
A score selects an action only through a declared policyUnder explicitly illustrative policy P7, score 0.62 falls between review thresholds 0.40 and 0.80 and therefore enters human review. The corrected decision retains policy version and reviewer provenance. Allowed-item audits and appeals also contribute corrected records.Illustrative policy P7; not universalthresholdsOne item receives score 0.6200.400.8010.40 <= 0.62 < 0.80: HUMAN REVIEWDecision + policy P7 + reviewerprovenanceAllowed-item audits and appeals also updatethe record.
In this illustrative policy, 0.62 enters human review, not automatic action. Audits and appeals preserve policy and reviewer provenance. These thresholds are not universal.
Check: How can a missed violation on the allow path become correction evidence?
Read it this way: Score 0.62 lies inside this illustrative P7 review region. It is not intrinsically a permission to publish or remove content.
Which P7 region contains 0.62?

Tells that get you a strong-hire vote

  • You separate policy from model.
  • You name per-policy thresholds rather than one global score.
  • You make human review and appeals part of the architecture.
  • You bring up adversarial robustness with continuous retraining.
  • You insist on per-slice fairness eval.
  • You name borderline routing as the high-value review queue.

Tells that get you down-leveled

  • One classifier, one threshold.
  • No human-in-the-loop or appeals process.
  • No mention of adversarial drift.
  • No fairness slicing.

Common follow-up

“How do you handle policy changes? E.g., the company updates its definition of hate speech.”

The L6 answer:

“Several layers. First, the policy spec is a versioned document; changes go through a review process and have an effective date. Second, retrain the model on data labeled under the new policy (re-label a held-out set first; if old labels are now misclassified, retrain; otherwise just patch the rules layer). Third, track ‘agreement with current policy’ over time as a metric; if the model drifts from policy faster than retraining can keep up, that’s a flag for accelerated retraining cadence. Fourth, communicate the change to moderators and users via release notes; appeals processed under the old policy may need re-review.”

Related: How do you handle hallucinations in production?, How do you deal with class imbalance in 2026?, A/B testing for ML systems.

Found an error or stale claim? Report a correction or send a private note.