Glenn Ceusters, Luis Ramirez Camargo, Rüdiger Franke, Ann Nowé, Maarten Messagie
Reinforcement learning (RL) is a promising optimal control technique for multi-energy management systems. It does not require a model a priori - reducing the upfront and ongoing project-specific engineering effort and is capable of learning better representations of the underlying system dynamics. However, vanilla RL does not provide constraint satisfaction guarantees — resulting in various potentially unsafe interactions within its environment. In this paper, we present two novel online model-free safe RL methods, namely SafeFallback and GiveSafe, where the safety constraint formulation is decoupled from the RL formulation. These provide hard-constraint satisfaction guarantees both during training and deployment of the (near) optimal policy. This is without the need of solving a mathematical program, resulting in less computational power requirements and more flexible constraint function formulations. In a simulated multi-energy systems case study we have shown that both methods start with a significantly higher utility compared to a vanilla RL benchmark and Optlayer benchmark (94,6\% and 82,8\% compared to 35,5\% and 77,8\%) and that the proposed SafeFallback method even can outperform the vanilla RL benchmark (102,9\% to 100\%). We conclude that both methods are viably safety constraint handling techniques applicable beyond RL, as demonstrated with random policies while still providing hard-constraint guarantees.
Ceusters, G, Camargo, LR, Franke, R, Nowé, A & Messagie, M 2023, 'Safe reinforcement learning for multi-energy management systems with known constraint functions', Energy and AI, vol. 12, 100227. https://doi.org/10.1016/j.egyai.2022.100227
Ceusters, G., Camargo, L. R., Franke, R., Nowé, A., & Messagie, M. (2023). Safe reinforcement learning for multi-energy management systems with known constraint functions. Energy and AI, 12, Article 100227. https://doi.org/10.1016/j.egyai.2022.100227
@article{10ca86774eaf4da8a9a1d1f61b26ca5a,
title = "Safe reinforcement learning for multi-energy management systems with known constraint functions",
abstract = "Reinforcement learning (RL) is a promising optimal control technique for multi-energy management systems. It does not require a model a priori - reducing the upfront and ongoing project-specific engineering effort and is capable of learning better representations of the underlying system dynamics. However, vanilla RL does not provide constraint satisfaction guarantees — resulting in various potentially unsafe interactions within its environment. In this paper, we present two novel online model-free safe RL methods, namely SafeFallback and GiveSafe, where the safety constraint formulation is decoupled from the RL formulation. These provide hard-constraint satisfaction guarantees both during training and deployment of the (near) optimal policy. This is without the need of solving a mathematical program, resulting in less computational power requirements and more flexible constraint function formulations. In a simulated multi-energy systems case study we have shown that both methods start with a significantly higher utility compared to a vanilla RL benchmark and Optlayer benchmark (94,6\% and 82,8\% compared to 35,5\% and 77,8\%) and that the proposed SafeFallback method even can outperform the vanilla RL benchmark (102,9\% to 100\%). We conclude that both methods are viably safety constraint handling techniques applicable beyond RL, as demonstrated with random policies while still providing hard-constraint guarantees.",
keywords = "Reinforcement Learning, Constraints, multi-energy systems, Energy Management Systems",
author = "Glenn Ceusters and Camargo, \{Luis Ramirez\} and R{\"u}diger Franke and Ann Now{\'e} and Maarten Messagie",
note = "Funding Information: This work has been supported in part by ABB n.v., Belgium and Flemish Agency for Innovation and Entrepreneurship (VLAIO) grant HBC.2019.2613 , Belgium. Publisher Copyright: {\textcopyright} 2022 The Author(s) Copyright: Copyright 2023 Elsevier B.V., All rights reserved.",
year = "2023",
month = apr,
doi = "10.1016/j.egyai.2022.100227",
language = "English",
volume = "12",
journal = "Energy and AI",
publisher = "Oxford: Elsevier Ltd.",
}