Summary
Joint evidence can help train local actors; it cannot supply facts unavailable when those actors execute.
Problem
Two cooperative agents choose A or B from separate observations. For one fixed state, their local action values are and . Compare additive and positive-weight team values, then test whether perfect self-play predicts success with an unfamiliar partner.
Centralized training, decentralized execution
Training may use global state and joint actions; deployed actors use only permitted local observations, history or declared messages. The drawing uses the simple observation-only case.
Learning objective
The centralized critic trains local policies but is absent during execution
Derivation
Can the deployed actor call the centralized training critic?
Calculate Local Choices and Team Values
Agent 1 greedily chooses A, while agent 2 chooses B.
| Joint choice | VDN: | Positive mixer: |
|---|---|---|
| A, A | 2 | 4 |
| A, B | 5 | 7 |
| B, A | 1 | 2 |
| B, B | 4 | 5 |
Both team functions are maximized by the independent choices here. Hold the state fixed and assume independent feasible action sets. Nonnegative mixer derivatives let local greedy choices attain a joint maximum:
QMIX uses this monotonicity constraint; it restricts representable team values. It does not identify each agent’s causal contribution or guarantee equilibrium.
Check a Convention Outside Self-Play
In a separate game, both agents receive one only when their symbols match. A learned actor always chooses A.
| Partner | Expected payoff |
|---|---|
| Self-play partner always A | 1 |
| Held-out partner equally A or B | 0.5 |
Perfect self-play did not establish partner generalization. This is an exact toy expectation, not an agent-training experiment.
Change 1: Unfamiliar or Adapting Agents
New task. Deployment partners or opponents differ from training.
Picture and change. Broaden populations and cross-play, including held-out conventions and exploiting policies.
Check. Stable self-play averages can conceal cycles or narrow conventions.
Use when. Training policies do not represent deployment.
Change 2: Communication Disappears
New task. A shared channel disappears or messages arrive late.
Picture and change. Restrict actor inputs, retrain permitted memory/communication and test missing/delayed messages.
Check. A privileged training critic cannot supply missing runtime information by itself.
Use when. The information boundary changes.
Failure: Training Privilege Leaks into Execution
An actor requiring unavailable global state violates decentralized execution. A team reward also does not automatically solve credit assignment or coordination.
Transfer Check
Must the joint transition law change merely because two agents exist?
Reasoned Answer
No. A stationary joint game can induce changing individual dynamics when other agents’ policies change.
Why it is harder than single-agent RL
Let be joint state, agent ‘s observation, and the joint action. Specify transition law , rewards , communication and policy versions. State which agents learn and whether incentives are cooperative, competitive or mixed.
| Challenge | Distinct question |
|---|---|
| Non-stationarity | Which other policies change the agent’s induced observations/transitions? |
| Credit | Which action contributed to shared return? |
| Partial observation | What local information/history is missing? |
| Coordination | Which conventions coexist? |
| Opponents | How can others strategically respond? |
| Evaluation | Which partners, seeds and exploiters are genuinely held out? |
Cooperative and competitive methods
VDN adds local values; QMIX conditions a monotonic mixer on training evidence. Neither represents every team payoff nor repairs missing local facts. MADDPG uses centralized action critics with decentralized actors. Self-play, leagues and population sampling broaden experience, not guarantee robustness or Nash equilibrium.
In an interview
- Name changing policies and incentives.
- Declare joint/local information and message permissions.
- Trace training versus deployment inputs.
- Check credit, coordination and communication failures.
- Report cross-play, seeds, adaptation and exploiting policies.
Common confusions
Multiple fixed policies do not automatically imply a changing joint transition law. Privileged critics, monotonic mixing and population generalization solve different problems.
Sources
Related: actor-critic methods, exploration versus exploitation, and PPO.