Summary
A value function scores actions; an actor produces actions or distributions. Choose data reuse and dynamics models separately.
Problem
For one state, actions are left, stay and right. Their return estimates are . An explicit actor instead gives probabilities . Trace each action path, then add epsilon-greedy behavior with and uniform exploration across all three actions.
Picture: Scores Need a Selection Rule
One state, two action interfaces
Values need an action-selection rule; that rule can still be stochastic
Derivation
Value: scores then a rule
Left: 1.2. Stay: 0.4. Right: 2.1.
Greedy selection chooses right. Epsilon-greedy instead chooses right with probability 13/15.
state → Q scores → rule → action
Bellman targets can train the values; the behavior rule need not be deterministic.
Policy: learn the selector
Left: 0.15. Stay: 0.10. Right: 0.75.
Sampling chooses right 75% of the time; using the mode always chooses right here.
state → actor → action
An actor can also be deterministic; a stochastic actor can use a return-weighted log-probability gradient.
Actor-critic: separate training path
transition → critic → baseline/target/actor derivative
A state/action critic supports learning; it need not choose deployed actions.
SAC combines a stochastic actor with off-policy Q learning, not an actor-equals-on-policy rule.
Which rule breaks the tie, and which rule preserves equal probability?
Check the Behavior Probabilities
For epsilon-greedy here, right receives the greedy branch plus its share of exploration:
Left and stay each receive . The actor’s right probability remains , despite both interfaces having the same mode. State behavior/target distributions and corrections explicitly when reusing data.
Change 1: Discrete Choices Become Continuous Torque
New task. Choose real-valued motor commands.
Picture and change. Price continuous Q maximization or use a bounded actor with controller/safety constraints.
Check. A small finite output head cannot enumerate every torque.
Use when. Argmax is no longer a cheap scan.
Change 2: Interaction Becomes Expensive
New task. Real-world samples are costly and the simulator is imperfect.
Picture and change. Compare reuse and learned-model planning with support, model-error, transfer and safe-fallback checks.
Check. Dynamics errors can compound; sample efficiency is not guaranteed.
Use when. Data cost changes the feasible methods.
Failure: An Algorithm Name Replaces Its Assumptions
A task-name table can hide unsupported offline actions, unsafe controls or missing observation memory. No universal variance or sample-efficiency ranking resolves those constraints.
Transfer Check
Is model-based learning mutually exclusive with actor-critic?
Reasoned Answer
No. Learned dynamics can support planning or training data alongside an actor and critic.
Value-based methods
Q-learning/DQN-family methods learn action values with chosen behavior and target rules. Replay, target networks and double estimators address specific data/stability risks, not unsupported offline actions automatically.
Policy-based methods
Actors may be stochastic or deterministic. PPO commonly reuses recent rollouts for several updates; SAC uses replay under its algorithm. Monte Carlo gradients can have high variance; baselines and critics introduce their own estimation considerations.
Actor-critic: combining the two
A critic can provide a state baseline, bootstrapped target or action derivative. SAC is stochastic/off-policy; TD3 uses a deterministic actor with Q critics.
Decision matrix
| Assumption | Candidate direction | Discriminating check |
|---|---|---|
| Modest finite actions | Enumerated Q values or actor | Selection/coverage cost |
| Continuous control | Actor or continuous optimizer | Bounds, stability, latency |
| Offline/costly data | Supported reuse or models | Coverage, shift, safe interaction |
| Partial observation | Memory/belief with either interface | Missing information and history |
| Preference fitting | Specified preference objective | DPO is not generic sequential online RL |
What about model-based RL?
Dynamics learning/planning is another axis, not an exclusive third interface. Dreamer/MuZero-style designs still need model-error and decision checks.
Common pitfalls
Do not equate actors with fresh-only data, values with determinism, or critics with guaranteed improvement.
Related
- Q-learning. Canonical value-based.
- Policy gradient. Canonical policy-based.
- PPO. Modern actor-critic.