Diagnosing Non-Intermittent Anomalies in Reinforcement Learning Policy Executions

Avraham Natan, Roni Stern, Meir Kalech

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Reinforcement learning (RL) algorithms output policies specifying which action an agent should take in a given state. However, faults can sometimes arise during policy execution due to internal faults in the agent. As a result, actions may have unexpected effects. In this work, we aim to diagnose such faults and infer their root cause. We consider two types of diagnosis problems. In the first, which we call RLDXw, we assume we only know what a normal execution looks like. In the second, called RLDXs, we assume we have models for the faulty behavior of a component, which we call fault modes. The solution to RLDXw is a time step at which a fault occurred for the first time. The solution to RLDXs is more informative, represented as a fault mode according to which the RL task was executed. Solving those problems is useful in practice to facilitate efficient repair of faulty agents, since it can focus the repair efforts on specific actions. We formally define RLDXw and RLDXs and design two algorithms called WFMa and SFMa for solving them. We evaluate our algorithms on a benchmark of RL domains and discuss their strengths and limitations. When the number of the observed states increases, both WFMa and SFMa report a decrease in runtime (up to significantly 6.5 times faster). Additionally, the runtime of SFMa increases linearly with the increase in candidate fault modes.

Original languageAmerican English
Title of host publication35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024
EditorsIngo Pill, Avraham Natan, Franz Wotawa
PublisherSchloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing
ISBN (Electronic)9783959773560
DOIs
StatePublished - 26 Nov 2024
Event35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024 - Vienna, Austria
Duration: 4 Nov 20247 Nov 2024

Publication series

NameOpenAccess Series in Informatics
Volume125

Conference

Conference35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024
Country/TerritoryAustria
CityVienna
Period4/11/247/11/24

Keywords

  • Autonomous Systems
  • Diagnosis
  • Reinforcement Learning

All Science Journal Classification (ASJC) codes

  • Geography, Planning and Development
  • Modelling and Simulation

Cite this