Despite many successful attempts at explaining Deep Rein- forcement Learning policies using distillation, it remains difficult to bal- ance the performance-interpretability trade-off and select a fitting surro- gatemodel.Inadditiontothis,traditionaldistillationonlyminimizesthe distance between the behavior of the original and the surrogate policy while other RL-specific components such as action value are disregarded. To solve this, we introduce a new model-agnostic method called Critic- Driven Voronoi State Partitioning, which partitions a black box control policy into regions where a simple class of model can be optimized using gradient descent. By exploiting the critic value network of the original policy, we iteratively introduce new subpolicies in regions with insuffi- cient value, standing in for a measure of policy complexity. The parti- tioning, a Voronoi quantizer, uses nearest neighbor lookups to assign a linear function to each point in the state space resulting in a cell-like diagram. We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.
Deproost, S, Nowe, A & Steckelmacher, D 2026, 'Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Interpretable Models', Lecture Notes in Computer Science.
Deproost, S., Nowe, A., & Steckelmacher, D. (Accepted/In press). Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Interpretable Models. Lecture Notes in Computer Science.
@article{a332275c07b14600a0f6e5a455efb4e7,
title = "Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Interpretable Models",
abstract = "Despite many successful attempts at explaining Deep Rein- forcement Learning policies using distillation, it remains difficult to bal- ance the performance-interpretability trade-off and select a fitting surro- gatemodel.Inadditiontothis,traditionaldistillationonlyminimizesthe distance between the behavior of the original and the surrogate policy while other RL-specific components such as action value are disregarded. To solve this, we introduce a new model-agnostic method called Critic- Driven Voronoi State Partitioning, which partitions a black box control policy into regions where a simple class of model can be optimized using gradient descent. By exploiting the critic value network of the original policy, we iteratively introduce new subpolicies in regions with insuffi- cient value, standing in for a measure of policy complexity. The parti- tioning, a Voronoi quantizer, uses nearest neighbor lookups to assign a linear function to each point in the state space resulting in a cell-like diagram. We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.",
keywords = "Reinforcement Learning, Explainable AI",
author = "Senne Deproost and Ann Nowe and Denis Steckelmacher",
year = "2026",
month = may,
day = "27",
language = "English",
journal = "Lecture Notes in Computer Science",
issn = "0302-9743",
publisher = "Springer Verlag",
}