Summary
Scalable oversight develops ways for humans and trusted systems to evaluate increasingly capable model work. AI control designs protocols that limit harm even if a powerful model may knowingly pursue an unwanted objective.
Both address a capability gap between the model and its overseer, but they make different assumptions.
Oversight often asks: how can a weaker evaluator reliably judge a stronger model’s answer, plan, proof, or code?
Control asks a harsher question: if the model may be actively untrusted, what combination of monitoring, restricted actions, trusted models, audits, and containment keeps expected harm below a threshold?
A better judge is not automatically a control protocol. A sandbox is not automatically an evaluation method.
Learning objective: given a proposed safety technique, identify whether it improves judgment under a capability gap, limits harm under intentional subversion, or contributes evidence to both.
Same capability gap, different contract
What must a safety technique accomplish when the stronger model may be merely hard to judge versus actively untrusted?
SHARED CHALLENGE · MODEL CAPABILITY EXCEEDS UNAIDED HUMAN EVALUATION
A human cannot simply redo every task.
Both fields can use decomposition, trusted models, monitors, tests, and selective human review, but the threat assumption determines what counts as success.
SCALABLE OVERSIGHT · MAKE HARD WORK JUDGEABLE
Threat assumption
The stronger model may be wrong, misleading, or difficult to evaluate; the method does not need to assume a deliberate attacker.
Intervention target
Increase evaluator competence with decomposition, debate, critique, recursive assistance, process evidence, or verification.
Characteristic failure
The evaluator accepts a bad answer because local checks do not compose, evidence is biased, or both systems share a blind spot.
Success test
On tasks beyond unaided human ability, the assisted evaluator more reliably distinguishes correct or acceptable work from failures.
AI CONTROL · CONTAIN AN INTENTIONAL ATTACKER
Threat assumption
The capable model may knowingly search for ways to cause harm while appearing benign to trusted components.
Intervention target
Constrain actions with least privilege, trusted monitoring, randomized audits, escalation, blocking, logging, and containment.
Characteristic failure
An adaptive attack finds a route through tools, side channels, monitor blind spots, or a predictable audit policy.
Success test
Under adaptive red-team strategies, the protocol keeps catastrophic failure acceptably low while retaining measured usefulness.
Scalable oversight methods
Decomposition
Break a task into subproblems a human or weaker model can judge. This works when local correctness composes into global correctness and the decomposition does not hide cross-cutting failures.
Debate or critique
Models produce arguments, critiques, or competing answers so a judge can identify errors. The mechanism relies on truth or error being easier to verify when surfaced than to produce from scratch.
Recursive assistance
Use model help to expand human evaluation capacity, such as summarizing evidence, generating tests, or tracing claims to sources. Assistance can also bias the evaluator or share the same blind spot.
Process supervision
Evaluate intermediate steps rather than only outcomes. This can provide denser signal but requires that steps are meaningful, observable, and difficult to game.
Verifiable environments
Use code execution, theorem checkers, simulators, or state transitions where correctness can be checked automatically. Verification covers only what the environment formalizes.
Control protocols
A simple control system might combine:
- an untrusted high-capability model;
- a trusted but weaker monitor;
- restricted tools and least privilege;
- randomized audits;
- escalation to human review;
- canary tasks that reveal suspicious behavior;
- logging and containment;
- a policy that decides when to use, inspect, or block the untrusted model.
Evaluation should estimate usefulness and safety under an adaptive red team. A protocol that blocks everything is safe but useless. A protocol that succeeds only against non-adaptive attacks is fragile.
Control evaluations
Define:
- threat model and untrusted-model capability;
- attack objective and budget;
- trusted components and their limitations;
- protocol and information available to each actor;
- safety failure and usefulness metrics;
- adaptive attack process;
- uncertainty and confidence in the estimated failure rate.
Rare catastrophic outcomes require careful statistical treatment. Zero observed failures in a small sample is not proof of negligible risk.
Common confusions
- “A stronger monitor solves oversight.” The target model can still exceed, manipulate, or share blind spots with the monitor.
- “Decomposition always helps.” Some failures live in interactions between subproblems.
- “Process supervision reveals intent.” Intermediate text can be incomplete or optimized for the supervisor.
- “Sandboxing is sufficient control.” Tools, data, humans, and side channels define a larger boundary.
- “A red team estimates real failure probability.” It demonstrates attacks under a search process; translating that to deployment risk requires reachability and prevalence assumptions.
- “Safety and usefulness collapse to one score.” A protocol’s trade-off frontier matters.
In an interview
State which problem you are solving: evaluator capability, untrusted-model control, or both. Define actors, information, protocol, adaptive attack, safety metric, usefulness metric, and the evidence needed to trust the result.
Related: chain-of-thought monitorability, model organisms of misalignment, and design an LLM red-team program.