Raphaël Avalos, Mathieu Reymond, Ann Nowe, Diederik M. Roijers
Multi-agent reinforcement learning (MARL) enables us to create adaptive agents in challenging environments, even when the agents have limited observation. Modern MARL methods have focused on finding factorized value functions. While successful, the resulting methods have convoluted network structures. We take a radically different approach and build on the structure of independent Q-learners. Our algorithm LAN leverages a dueling architecture to represent decentralized policies as separate individual advantage functions w.r.t.\ a centralized critic that is cast aside after training. The critic works as a stabilizer that coordinates the learning and to formulate DQN targets. This enables LAN to keep the number of parameters of its centralized network independent in the number of agents, without imposing additional constraints like monotonic value functions. When evaluated on the SMAC, LAN shows SOTA performance overall and scores more than 80\% wins in two super-hard maps where even QPLEX does not obtain almost any wins. Moreover, when the number of agents becomes large, LAN uses significantly fewer parameters than QPLEX or even QMIX. We thus show that LAN's structure forms a key improvement that helps MARL methods remain scalable.
Avalos, R, Reymond, M, Nowe, A & Roijers, DM 2022, Local Advantage Networks for Cooperative Multi-Agent Reinforcement Learning. in International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022: Extended Abstract. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, vol. 3, IFAAMAS, pp. 1524-1526, 21st International Conference on Autonomous Agents and Multi-agent System, 9/05/22.
Avalos, R., Reymond, M., Nowe, A., & Roijers, D. M. (2022). Local Advantage Networks for Cooperative Multi-Agent Reinforcement Learning. In International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022: Extended Abstract (pp. 1524-1526). (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS; Vol. 3). IFAAMAS.
@inproceedings{74ed67672d214e93a9b8f99bf162a9f2,
title = "Local Advantage Networks for Cooperative Multi-Agent Reinforcement Learning",
abstract = "Multi-agent reinforcement learning (MARL) enables us to create adaptive agents in challenging environments, even when the agents have limited observation. Modern MARL methods have focused on finding factorized value functions. While successful, the resulting methods have convoluted network structures. We take a radically different approach and build on the structure of independent Q-learners. Our algorithm LAN leverages a dueling architecture to represent decentralized policies as separate individual advantage functions w.r.t.\ a centralized critic that is cast aside after training. The critic works as a stabilizer that coordinates the learning and to formulate DQN targets. This enables LAN to keep the number of parameters of its centralized network independent in the number of agents, without imposing additional constraints like monotonic value functions. When evaluated on the SMAC, LAN shows SOTA performance overall and scores more than 80\% wins in two super-hard maps where even QPLEX does not obtain almost any wins. Moreover, when the number of agents becomes large, LAN uses significantly fewer parameters than QPLEX or even QMIX. We thus show that LAN's structure forms a key improvement that helps MARL methods remain scalable.",
author = "Rapha{\"e}l Avalos and Mathieu Reymond and Ann Nowe and Roijers, {Diederik M.}",
note = "Funding Information: Rapha{\"e}l Avalos was supported by the FWO (grant 11F5721N). This research was supported by the Flemish Government under the “Onderzoeksprogramma Artifici{\"e}le Intelligentie (AI) Vlaanderen” program. Publisher Copyright: {\textcopyright} 2022 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved. Copyright: Copyright 2022 Elsevier B.V., All rights reserved.; 21st International Conference on Autonomous Agents and Multi-agent System, AAMAS ; Conference date: 09-05-2022 Through 13-05-2022",
year = "2022",
month = may,
day = "9",
language = "English",
series = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",
publisher = "IFAAMAS",
pages = "1524--1526",
booktitle = "International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022",
url = "https://aamas2022-conference.auckland.ac.nz",
}