Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents

Explainable reinforcement learning methods can roughly be divided into local explanations that analyze specific decisions of the agents and global explanations that convey the general strategy of the agents. In this work, we study a novel combination of local and global explanations for reinforcement learning agents. Specifically, we combine reward decomposition, a local explanation method that exposes which components of the reward function influenced a specific decision, and HIGHLIGHTS, a global explanation method that shows a summary of the agent’s behavior in decisive states. Results from two user studies show significant benefits for both methods. We found that the local reward decomposition was more useful for identifying the agents’ priorities. However, when there was only a minor difference between the agents’ preferences, the global information provided by HIGHLIGHTS additionally improved participants’ understanding.