TY - GEN
T1 - Diagnosing Non-Intermittent Anomalies in Reinforcement Learning Policy Executions
AU - Natan, Avraham
AU - Stern, Roni
AU - Kalech, Meir
N1 - Publisher Copyright: © Avraham Natan, Roni Stern, and Meir Kalech.
PY - 2024/11/26
Y1 - 2024/11/26
N2 - Reinforcement learning (RL) algorithms output policies specifying which action an agent should take in a given state. However, faults can sometimes arise during policy execution due to internal faults in the agent. As a result, actions may have unexpected effects. In this work, we aim to diagnose such faults and infer their root cause. We consider two types of diagnosis problems. In the first, which we call RLDXw, we assume we only know what a normal execution looks like. In the second, called RLDXs, we assume we have models for the faulty behavior of a component, which we call fault modes. The solution to RLDXw is a time step at which a fault occurred for the first time. The solution to RLDXs is more informative, represented as a fault mode according to which the RL task was executed. Solving those problems is useful in practice to facilitate efficient repair of faulty agents, since it can focus the repair efforts on specific actions. We formally define RLDXw and RLDXs and design two algorithms called WFMa and SFMa for solving them. We evaluate our algorithms on a benchmark of RL domains and discuss their strengths and limitations. When the number of the observed states increases, both WFMa and SFMa report a decrease in runtime (up to significantly 6.5 times faster). Additionally, the runtime of SFMa increases linearly with the increase in candidate fault modes.
AB - Reinforcement learning (RL) algorithms output policies specifying which action an agent should take in a given state. However, faults can sometimes arise during policy execution due to internal faults in the agent. As a result, actions may have unexpected effects. In this work, we aim to diagnose such faults and infer their root cause. We consider two types of diagnosis problems. In the first, which we call RLDXw, we assume we only know what a normal execution looks like. In the second, called RLDXs, we assume we have models for the faulty behavior of a component, which we call fault modes. The solution to RLDXw is a time step at which a fault occurred for the first time. The solution to RLDXs is more informative, represented as a fault mode according to which the RL task was executed. Solving those problems is useful in practice to facilitate efficient repair of faulty agents, since it can focus the repair efforts on specific actions. We formally define RLDXw and RLDXs and design two algorithms called WFMa and SFMa for solving them. We evaluate our algorithms on a benchmark of RL domains and discuss their strengths and limitations. When the number of the observed states increases, both WFMa and SFMa report a decrease in runtime (up to significantly 6.5 times faster). Additionally, the runtime of SFMa increases linearly with the increase in candidate fault modes.
KW - Autonomous Systems
KW - Diagnosis
KW - Reinforcement Learning
UR - http://www.scopus.com/inward/record.url?scp=85211909777&partnerID=8YFLogxK
U2 - https://doi.org/10.4230/OASIcs.DX.2024.23
DO - https://doi.org/10.4230/OASIcs.DX.2024.23
M3 - Conference contribution
T3 - OpenAccess Series in Informatics
BT - 35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024
A2 - Pill, Ingo
A2 - Natan, Avraham
A2 - Wotawa, Franz
PB - Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing
T2 - 35th International Conference on Principles of Diagnosis and Resilient Systems, DX 2024
Y2 - 4 November 2024 through 7 November 2024
ER -