Formally Explaining Neural Networks within Reactive Systems

Shahaf Bassan, Guy Amir, Davide Corsi, Idan Refaeli, Guy Katz

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Deep neural networks (DNNs) are increasingly being used as controllers in reactive systems. However, DNNs are highly opaque, which renders it difficult to explain and justify their actions. To mitigate this issue, there has been a surge of interest in explainable AI (XAI) techniques, capable of pinpointing the input features that caused the DNN to act as it did. Existing XAI techniques typically face two limitations: (i) they are heuristic, and do not provide formal guarantees that the explanations are correct; and (ii) they often apply to 'one-shot' systems, where the DNN is invoked independently of past invocations, as opposed to reactive systems. Here, we begin bridging this gap, and propose a formal DNN-verification-based XAI technique for reasoning about multi-step, reactive systems. We suggest methods for efficiently calculating succinct explanations, by exploiting the system's transition constraints in order to curtail the search space explored by the underlying verifier. We evaluate our approach on two popular benchmarks from the domain of automated navigation; and observe that our methods allow the efficient computation of minimal and minimum explanations, significantly outperforming the state of the art. We also demonstrate that our methods produce formal explanations that are more reliable than competing, non-verification-based XAI techniques.

Original languageAmerican English
Title of host publicationProceedings of the 23rd Conference on Formal Methods in Computer-Aided Design, FMCAD 2023
EditorsAlexander Nadel, Kristin Yvonne Rozier, Warren A. Hunt, Georg Weissenbacher
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages10-22
Number of pages13
ISBN (Electronic)9783854480600
DOIs
StatePublished - 2023
Event23rd International Conference on Formal Methods in Computer-Aided Design, FMCAD 2023 - Ames, United States
Duration: 24 Oct 202327 Oct 2023

Publication series

NameProceedings of the 23rd Conference on Formal Methods in Computer-Aided Design, FMCAD 2023

Conference

Conference23rd International Conference on Formal Methods in Computer-Aided Design, FMCAD 2023
Country/TerritoryUnited States
CityAmes
Period24/10/2327/10/23

All Science Journal Classification (ASJC) codes

  • Computer Science (miscellaneous)
  • Artificial Intelligence
  • Computer Graphics and Computer-Aided Design
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Formally Explaining Neural Networks within Reactive Systems'. Together they form a unique fingerprint.

Cite this