Parallelization Techniques for Verifying Neural Networks

Haoze Wu; Alex Ozdemir; Aleksandar Zeljic; Kyle Julian; Ahmed Irfan; Divya Gopinath; Sadjad Fouladi; Guy Katz; Corina Pasareanu; Clark Barrett

doi:10.34727/2020/isbn.978-3-85448-042-6_20

Parallelization Techniques for Verifying Neural Networks

Haoze Wu
, Alex Ozdemir
, Aleksandar Zeljic
, Kyle Julian
, Ahmed Irfan
, Divya Gopinath
, Sadjad Fouladi
, Guy Katz
, Corina Pasareanu
, Clark Barrett

The Rachel and Selim Benin School of Engineering and Computer Science

The Hebrew University of Jerusalem

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Inspired by recent successes of parallel techniques for solving Boolean satisfiability, we investigate a set of strategies and heuristics to leverage parallelism and improve the scalability of neural network verification. We present a general description of the Split-and-Conquer partitioning algorithm, implemented within the Marabou framework, and discuss its parameters and heuristic choices. In particular, we explore two novel partitioning strategies, that partition the input space or the phases of the neuron activations, respectively. We introduce a branching heuristic and a direction heuristic that are based on the notion of polarity. We also introduce a highly parallelizable pre-processing algorithm for simplifying neural network verification problems. An extensive experimental evaluation shows the benefit of these techniques on both existing and new benchmarks. A preliminary experiment ultra-scaling our algorithm using a large distributed cloud-based platform also shows promising results.

Original language	English
Title of host publication	Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020
Editors	Alexander Ivrii, Ofer Strichman, Warren A. Hunt, Georg Weissenbacher
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	128-137
Number of pages	10
ISBN (Electronic)	9783854480426
DOIs	https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_20
State	Published - 21 Sep 2020
Event	20th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2020 - Virtual, Haifa, Israel Duration: 21 Sep 2020 → 24 Sep 2020

Publication series

Name	Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020

Conference

Conference	20th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2020
Country/Territory	Israel
City	Virtual, Haifa
Period	21/09/20 → 24/09/20

ASJC Scopus subject areas

Artificial Intelligence
Computer Graphics and Computer-Aided Design
Software
Safety, Risk, Reliability and Quality
Modelling and Simulation

Access to Document

10.34727/2020/isbn.978-3-85448-042-6_20

Cite this

Wu, H., Ozdemir, A., Zeljic, A., Julian, K., Irfan, A., Gopinath, D., Fouladi, S., Katz, G., Pasareanu, C., & Barrett, C. (2020). Parallelization Techniques for Verifying Neural Networks. In A. Ivrii, O. Strichman, W. A. Hunt, & G. Weissenbacher (Eds.), Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020 (pp. 128-137). Article 9283631 (Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_20

Wu, Haoze ; Ozdemir, Alex ; Zeljic, Aleksandar et al. / Parallelization Techniques for Verifying Neural Networks. Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020. editor / Alexander Ivrii ; Ofer Strichman ; Warren A. Hunt ; Georg Weissenbacher. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 128-137 (Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020).

@inproceedings{dccb42f7dd454f0497565cb990d4176b,

title = "Parallelization Techniques for Verifying Neural Networks",

abstract = "Inspired by recent successes of parallel techniques for solving Boolean satisfiability, we investigate a set of strategies and heuristics to leverage parallelism and improve the scalability of neural network verification. We present a general description of the Split-and-Conquer partitioning algorithm, implemented within the Marabou framework, and discuss its parameters and heuristic choices. In particular, we explore two novel partitioning strategies, that partition the input space or the phases of the neuron activations, respectively. We introduce a branching heuristic and a direction heuristic that are based on the notion of polarity. We also introduce a highly parallelizable pre-processing algorithm for simplifying neural network verification problems. An extensive experimental evaluation shows the benefit of these techniques on both existing and new benchmarks. A preliminary experiment ultra-scaling our algorithm using a large distributed cloud-based platform also shows promising results.",

author = "Haoze Wu and Alex Ozdemir and Aleksandar Zeljic and Kyle Julian and Ahmed Irfan and Divya Gopinath and Sadjad Fouladi and Guy Katz and Corina Pasareanu and Clark Barrett",

note = "Publisher Copyright: {\textcopyright} 2020 FMCAD Association.; 20th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2020 ; Conference date: 21-09-2020 Through 24-09-2020",

year = "2020",

month = sep,

day = "21",

doi = "10.34727/2020/isbn.978-3-85448-042-6\_20",

language = "English",

series = "Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "128--137",

editor = "Alexander Ivrii and Ofer Strichman and Hunt, \{Warren A.\} and Georg Weissenbacher",

booktitle = "Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020",

address = "United States",

}

Wu, H, Ozdemir, A, Zeljic, A, Julian, K, Irfan, A, Gopinath, D, Fouladi, S, Katz, G, Pasareanu, C & Barrett, C 2020, Parallelization Techniques for Verifying Neural Networks. in A Ivrii, O Strichman, WA Hunt & G Weissenbacher (eds), Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020., 9283631, Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020, Institute of Electrical and Electronics Engineers Inc., pp. 128-137, 20th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2020, Virtual, Haifa, Israel, 21/09/20. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_20

Parallelization Techniques for Verifying Neural Networks. / Wu, Haoze; Ozdemir, Alex; Zeljic, Aleksandar et al.
Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020. ed. / Alexander Ivrii; Ofer Strichman; Warren A. Hunt; Georg Weissenbacher. Institute of Electrical and Electronics Engineers Inc., 2020. p. 128-137 9283631 (Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Parallelization Techniques for Verifying Neural Networks

AU - Wu, Haoze

AU - Ozdemir, Alex

AU - Zeljic, Aleksandar

AU - Julian, Kyle

AU - Irfan, Ahmed

AU - Gopinath, Divya

AU - Fouladi, Sadjad

AU - Katz, Guy

AU - Pasareanu, Corina

AU - Barrett, Clark

PY - 2020/9/21

Y1 - 2020/9/21

N2 - Inspired by recent successes of parallel techniques for solving Boolean satisfiability, we investigate a set of strategies and heuristics to leverage parallelism and improve the scalability of neural network verification. We present a general description of the Split-and-Conquer partitioning algorithm, implemented within the Marabou framework, and discuss its parameters and heuristic choices. In particular, we explore two novel partitioning strategies, that partition the input space or the phases of the neuron activations, respectively. We introduce a branching heuristic and a direction heuristic that are based on the notion of polarity. We also introduce a highly parallelizable pre-processing algorithm for simplifying neural network verification problems. An extensive experimental evaluation shows the benefit of these techniques on both existing and new benchmarks. A preliminary experiment ultra-scaling our algorithm using a large distributed cloud-based platform also shows promising results.

AB - Inspired by recent successes of parallel techniques for solving Boolean satisfiability, we investigate a set of strategies and heuristics to leverage parallelism and improve the scalability of neural network verification. We present a general description of the Split-and-Conquer partitioning algorithm, implemented within the Marabou framework, and discuss its parameters and heuristic choices. In particular, we explore two novel partitioning strategies, that partition the input space or the phases of the neuron activations, respectively. We introduce a branching heuristic and a direction heuristic that are based on the notion of polarity. We also introduce a highly parallelizable pre-processing algorithm for simplifying neural network verification problems. An extensive experimental evaluation shows the benefit of these techniques on both existing and new benchmarks. A preliminary experiment ultra-scaling our algorithm using a large distributed cloud-based platform also shows promising results.

UR - https://www.scopus.com/pages/publications/85097996932

U2 - 10.34727/2020/isbn.978-3-85448-042-6_20

DO - 10.34727/2020/isbn.978-3-85448-042-6_20

M3 - Conference contribution

T3 - Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020

SP - 128

EP - 137

BT - Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020

A2 - Ivrii, Alexander

A2 - Strichman, Ofer

A2 - Hunt, Warren A.

A2 - Weissenbacher, Georg

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 20th International Conference on Formal Methods in Computer-Aided Design, FMCAD 2020

Y2 - 21 September 2020 through 24 September 2020

ER -

Wu H, Ozdemir A, Zeljic A, Julian K, Irfan A, Gopinath D et al. Parallelization Techniques for Verifying Neural Networks. In Ivrii A, Strichman O, Hunt WA, Weissenbacher G, editors, Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 128-137. 9283631. (Proceedings of the 20th Conference on Formal Methods in Computer-Aided Design, FMCAD 2020). doi: 10.34727/2020/isbn.978-3-85448-042-6_20

Parallelization Techniques for Verifying Neural Networks

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this