We introduce the Laser Learning Environment (LLE), a collaborative multi-agent reinforcement learning environment where coordination is key. In LLE, agents depend on each other to make progress (interdependence), must jointly take specific sequences of actions to succeed (perfect coordination), and accomplishing those joint actions does not yield any intermediate reward (zero-incentive dynamics). The challenge of such problems lies in the difficulty of escaping state space bottlenecks caused by interdependence steps since escaping those bottlenecks is not rewarded. We test multiple state-of-the-art value-based MARL algorithms against LLE and show that they consistently fail at the collaborative task because of their inability to escape state space bottlenecks, even though they successfully achieve perfect coordination. We show that Q-learning extensions such as prioritised experience replay and n-steps return hinder exploration in environments with zero-incentive dynamics, and find that intrinsic curiosity with random network distillation is not sufficient to escape those bottlenecks. We demonstrate the need for novel methods to solve this problem and the relevance of LLE as cooperative MARL benchmark.
Molinghen, Y, Avalos, R, Van Achter, M, Nowé, A & Lenaerts, T 2025, Laser Learning Environment: A New Environment for Coordination-Critical Multi-agent Tasks. in FA Oliehoek, M Kok & S Verwer (eds), Artificial Intelligence and Machine Learning - 35th Benelux Conference, BNAIC/Benelearn 2023, Revised Selected Papers. Communications in Computer and Information Science, vol. 2187 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 135-154, 35th Benelux Conference on Artificial Intelligence and Machine Learning, BNAIC/Benelearn 2023, Delft, Netherlands, 8/11/23. https://doi.org/10.1007/978-3-031-74650-5_8
Molinghen, Y., Avalos, R., Van Achter, M., Nowé, A., & Lenaerts, T. (2025). Laser Learning Environment: A New Environment for Coordination-Critical Multi-agent Tasks. In F. A. Oliehoek, M. Kok, & S. Verwer (Eds.), Artificial Intelligence and Machine Learning - 35th Benelux Conference, BNAIC/Benelearn 2023, Revised Selected Papers (pp. 135-154). (Communications in Computer and Information Science; Vol. 2187 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-74650-5_8
@inproceedings{7943c2d1f0a0488488d7efbc0362bbfb,
title = "Laser Learning Environment: A New Environment for Coordination-Critical Multi-agent Tasks",
abstract = "We introduce the Laser Learning Environment (LLE), a collaborative multi-agent reinforcement learning environment where coordination is key. In LLE, agents depend on each other to make progress (interdependence), must jointly take specific sequences of actions to succeed (perfect coordination), and accomplishing those joint actions does not yield any intermediate reward (zero-incentive dynamics). The challenge of such problems lies in the difficulty of escaping state space bottlenecks caused by interdependence steps since escaping those bottlenecks is not rewarded. We test multiple state-of-the-art value-based MARL algorithms against LLE and show that they consistently fail at the collaborative task because of their inability to escape state space bottlenecks, even though they successfully achieve perfect coordination. We show that Q-learning extensions such as prioritised experience replay and n-steps return hinder exploration in environments with zero-incentive dynamics, and find that intrinsic curiosity with random network distillation is not sufficient to escape those bottlenecks. We demonstrate the need for novel methods to solve this problem and the relevance of LLE as cooperative MARL benchmark.",
keywords = "Cooperative, Multi-Agent, Reinforcement Learning",
author = "Yannick Molinghen and Rapha{\"e}l Avalos and {Van Achter}, Mark and Ann Now{\'e} and Tom Lenaerts",
note = "Funding Information: Rapha\u00EBl Avalos is supported by the FWO (Research Foundation \u2013 Flanders) under the\u00A0grant 11F5721N. Tom Lenaerts is supported by an FWO project (grant nr. G054919N) and two FRS-FNRS PDR (grant numbers 31257234\u00A0and 40007793). His is furthermore supported by Service Public\u00A0de Wallonie Recherche under grant no 2010235-ariac\u00A0by digitalwallonia4.ai. Ann Now\u00E9 and Tom Lenaerts are also suported by the Flemish Government through the AI Research Program\u00A0and TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 35th Benelux Conference on Artificial Intelligence and Machine Learning, BNAIC/Benelearn 2023 ; Conference date: 08-11-2023 Through 10-11-2023",
year = "2025",
doi = "10.1007/978-3-031-74650-5_8",
language = "English",
isbn = "9783031746499",
series = "Communications in Computer and Information Science",
publisher = "Springer Science and Business Media Deutschland GmbH",
pages = "135--154",
editor = "Oliehoek, {Frans A.} and Manon Kok and Sicco Verwer",
booktitle = "Artificial Intelligence and Machine Learning - 35th Benelux Conference, BNAIC/Benelearn 2023, Revised Selected Papers",
address = "Germany",
}