Skip to content
mlmentorship

Multi-agent reinforcement learning

Learning when other agents change the environment: non-stationarity, credit assignment, coordination, competition, and evaluation.

Published · 4 min read ·Specialist ·Advanced

Visual quick review

Visual first · depth when needed

Separate the global state and joint actions available to a centralized learner during training from the local observation available to each policy during decentralized execution.

Preparing the visual…

Summary

Single-agent RL assumes a stationary environment. The moment other learning agents share it, that assumption breaks: markets, game-playing, traffic, negotiation, ad auctions, and self-play training are all multi-agent. Multi-agent RL studies environments where several agents act at once, and each agent’s reward and transitions depend on the other agents’ policies, which are themselves changing.

Why it is harder than single-agent RL

  • Non-stationarity: from one agent’s view, the environment shifts as the others learn.
  • Credit assignment: a shared team reward does not say whose action helped.
  • Partial observability: agents usually see only local information.
  • Coordination equilibria: several stable conventions can coexist.
  • Opponent modeling: competitive agents adapt strategically to you.
  • Evaluation: doing well against one set of opponents need not generalize.

Centralized training, decentralized execution

The common pattern trains critics or value functions with global state and all agents’ actions, while each deployed policy acts on its local observation only. MADDPG and value-decomposition methods apply this idea differently.

Learning objective

Separate information used to learn coordinated policies from information each policy can use when acting.

Centralized training and decentralized execution information paths During training, agent one and agent two each map a local observation to an action. A centralized critic additionally receives global state and both actions, then sends dashed learning signals back to both policies. During execution, the critic and global state are absent. Each trained policy independently maps only its own local observation to its own action. TRAINING · extra information may shape the update Agent 1 seeslocal o₁ Policy π₁parameters θ₁ Actiona₁ Agent 2 seeslocal o₂ Policy π₂parameters θ₂ Actiona₂ Global states Centralized criticQᵢ(s, a₁, a₂) dashed: learning signal updates policies during training EXECUTION · the critic and global state are not inputs local o₁ trained π₁uses o₁ only a₁ local o₂ trained π₂uses o₂ only a₂
Read it this way: read the solid paths first: each actor always chooses from its own observation. During training only, the critic may inspect global state and the joint action, then send dashed learning signals into both policies. Below the divider, that privileged path is gone; execution remains decentralized. Original schematic checked against MADDPG and QMIX.

Cooperative and competitive methods

Cooperative. Value decomposition builds a team value from per-agent values: VDN sums them; QMIX uses a monotonic mixing network so decentralized greedy actions stay consistent with the joint value.

Competitive and mixed. Self-play, population-based training, opponent sampling, and league systems reduce overfitting to a single opponent. Nash equilibrium is a useful reference point but hard to compute in large stochastic games.

In an interview

  1. Name the source of non-stationarity.
  2. Clarify cooperative, competitive, or mixed incentives.
  3. Explain centralized training with decentralized execution.
  4. Address credit assignment and communication.
  5. Evaluate against diverse policies, unseen partners, and exploiters.

Common confusions

  • “Just treat other agents as part of the environment.” Their policies change and respond strategically.
  • “Self-play guarantees robustness.” It can cycle or overfit to its own population.
  • “A team reward creates teamwork.” It can also create free-riding and muddy credit assignment.

Related: actor-critic methods, exploration versus exploitation, and PPO.

Found an error or stale claim? Report a correction or send a private note.