In multi-objective optimization, learning all the policies that reach Pareto-efficient solutions is an expensive process. The set of optimal policies can grow exponentially with the number of objectives, and recovering all solutions requires an exhaustive exploration of the entire state space. We propose Pareto Conditioned Networks (PCN), a method that uses a single neural network to encompass all non-dominated policies. PCN associates every past transition with its episode's return. It trains the network such that, when conditioned on this same return, it should reenact said transition. In doing so we transform the optimization problem into a classification problem. We recover a concrete policy by conditioning the network on the desired Pareto-efficient solution. Our method is stable as it learns in a supervised fashion, thus avoiding moving target issues. Moreover, by using a single network, PCN scales efficiently with the number of objectives. Finally, it makes minimal assumptions on the shape of the Pareto front, which makes it suitable to a wider range of problems than previous state-of-the-art multi-objective reinforcement learning algorithms.
Reymond, M, Bargiacchi, E & Nowe, A 2022, Pareto Conditioned Networks. in The 21st International Conference on Autonomous Agents and Multiagent Systems. IFAAMAS, pp. 1110-1118, 21st International Conference on Autonomous Agents and Multi-agent System, 9/05/22.
Reymond, M., Bargiacchi, E., & Nowe, A. (2022). Pareto Conditioned Networks. In The 21st International Conference on Autonomous Agents and Multiagent Systems (pp. 1110-1118). IFAAMAS.
@inproceedings{9c532c39109847e8a30facf8dc80bebb,
title = "Pareto Conditioned Networks",
abstract = "In multi-objective optimization, learning all the policies that reach Pareto-efficient solutions is an expensive process. The set of optimal policies can grow exponentially with the number of objectives, and recovering all solutions requires an exhaustive exploration of the entire state space. We propose Pareto Conditioned Networks (PCN), a method that uses a single neural network to encompass all non-dominated policies. PCN associates every past transition with its episode's return. It trains the network such that, when conditioned on this same return, it should reenact said transition. In doing so we transform the optimization problem into a classification problem. We recover a concrete policy by conditioning the network on the desired Pareto-efficient solution. Our method is stable as it learns in a supervised fashion, thus avoiding moving target issues. Moreover, by using a single network, PCN scales efficiently with the number of objectives. Finally, it makes minimal assumptions on the shape of the Pareto front, which makes it suitable to a wider range of problems than previous state-of-the-art multi-objective reinforcement learning algorithms.",
author = "Mathieu Reymond and Eugenio Bargiacchi and Ann Nowe",
note = "Funding Information: The authors would like to acknowledge FWO (Fonds Wetenschappelijk Onderzoek) for their support through the SB grant of Eugenio Bargiacchi (#1SA2820N). This research was additionally supported by funding from the Flemish Government under the âOnderzoeksprogramma Artifici{\"e}le Intelligentie (AI) Vlaanderenâ programme. We would also like to thank Diederik M. Roijers for helpful feedback. Funding Information: by funding from the Flemish Government under the âOnderzoek-sprogramma Artifici{\"e}le Intelligentie (AI) Vlaanderenâ programme. Funding Information: The authors would like to acknowledge FWO (Fonds Wetenschap-pelijk Onderzoek) for their support through the SB grant of Eugenio Bargiacchi (#1SA2820N). This research was additionally supported Publisher Copyright: {\textcopyright} 2022 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved Copyright: Copyright 2022 Elsevier B.V., All rights reserved.; 21st International Conference on Autonomous Agents and Multi-agent System, AAMAS ; Conference date: 09-05-2022 Through 13-05-2022",
year = "2022",
month = may,
day = "9",
language = "English",
pages = "1110--1118",
booktitle = "The 21st International Conference on Autonomous Agents and Multiagent Systems",
publisher = "IFAAMAS",
url = "https://aamas2022-conference.auckland.ac.nz",
}