Roxana Radulescu, Timothy Verstraeten, Yijie Zhang, Patrick Mannion, Diederik M. Roijers, Ann Nowe
Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with nonlinear utilities. Specifically, we consider two-player multi-objective normal form games with nonlinear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e. learning while considering the impact of one{\textquoteright}s policy when anticipating the opponent{\textquoteright}s learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.
Radulescu, R, Verstraeten, T, Zhang, Y, Mannion, P, Roijers, DM & Nowe, A 2021, 'Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games', Neural Computing & Applications, vol. 34, no. 3, pp. 1759-1781. https://doi.org/10.1007/s00521-021-06184-3
Radulescu, R., Verstraeten, T., Zhang, Y., Mannion, P., Roijers, D. M., & Nowe, A. (2021). Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games. Neural Computing & Applications, 34(3), 1759-1781. https://doi.org/10.1007/s00521-021-06184-3
@article{1d4a35b0cc08418c9a81f9875a798a67,
title = "Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games",
abstract = "Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with nonlinear utilities. Specifically, we consider two-player multi-objective normal form games with nonlinear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e. learning while considering the impact of one{\textquoteright}s policy when anticipating the opponent{\textquoteright}s learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.",
author = "Roxana Radulescu and Timothy Verstraeten and Yijie Zhang and Patrick Mannion and Roijers, {Diederik M.} and Ann Nowe",
note = "Funding Information: The authors would like to acknowledge FWO (Fonds Wetenschappelijk Onderzoek) for their support through the SB grants of Timothy Verstraeten (#1S47617N). This research was supported by funding from the Flemish Government under the âOnderzoeksprogramma Artifici{\"e}le Intelligentie (AI) Vlaanderenâ program. Publisher Copyright: {\textcopyright} 2021, The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature.; 2020 Adaptive Learning Agents workshop at AAMAS, ALA 2020 ; Conference date: 09-05-2020 Through 10-05-2020",
year = "2021",
month = jun,
day = "19",
doi = "10.1007/s00521-021-06184-3",
language = "English",
volume = "34",
pages = "1759--1781",
journal = "Neural Computing & Applications",
issn = "0941-0643",
publisher = "Springer London",
number = "3",
url = "https://ala2020.vub.ac.be",
}