Evolution of activation functions for deep learning-based image classification.

Raz Lapid; Moshe Sipper

doi:10.1145/3520304.3533949

Evolution of activation functions for deep learning-based image classification.

Raz Lapid, Moshe Sipper

Department of Computer Science

Ben-Gurion University of the Negev

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

Original language	American English
Title of host publication	Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022
Place of Publication	New York, NY, USA
Pages	2113-2121
Number of pages	9
ISBN (Electronic)	9781450392686
DOIs	https://doi.org/10.1145/3520304.3533949
State	Published - 19 Jul 2022
Event	2022 Genetic and Evolutionary Computation Conference, GECCO 2022 - Virtual, Online, United States Duration: 9 Jul 2022 → 13 Jul 2022

Conference

Conference	2022 Genetic and Evolutionary Computation Conference, GECCO 2022
Country/Territory	United States
City	Virtual, Online
Period	9/07/22 → 13/07/22

Keywords

Computer graphics
Computing methodologies
Design and analysis of algorithms
Discrete optimization
Image manipulation
Image processing
Machine learning
Mathematical optimization
Optimization with randomized search heuristics
Theory of computation

Access to Document

10.1145/3520304.3533949

https://dblp.org/db/conf/gecco/gecco2022c.html#LapidS22

Cite this

@inproceedings{e7c77435261748cabfa8da0ede35562f,

title = "Evolution of activation functions for deep learning-based image classification.",

abstract = "Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.",

keywords = "Computer graphics, Computing methodologies, Design and analysis of algorithms, Discrete optimization, Image manipulation, Image processing, Machine learning, Mathematical optimization, Optimization with randomized search heuristics, Theory of computation",

author = "Raz Lapid and Moshe Sipper",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.; 2022 Genetic and Evolutionary Computation Conference, GECCO 2022 ; Conference date: 09-07-2022 Through 13-07-2022",

year = "2022",

month = jul,

day = "19",

doi = "10.1145/3520304.3533949",

language = "American English",

pages = "2113--2121",

booktitle = "Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022",

}

Lapid, R & Sipper, M 2022, Evolution of activation functions for deep learning-based image classification. in Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022. New York, NY, USA, pp. 2113-2121, 2022 Genetic and Evolutionary Computation Conference, GECCO 2022, Virtual, Online, United States, 9/07/22. https://doi.org/10.1145/3520304.3533949

TY - GEN

T1 - Evolution of activation functions for deep learning-based image classification.

AU - Lapid, Raz

AU - Sipper, Moshe

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2022/7/19

Y1 - 2022/7/19

N2 - Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

AB - Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

KW - Computer graphics

KW - Computing methodologies

KW - Design and analysis of algorithms

KW - Discrete optimization

KW - Image manipulation

KW - Image processing

KW - Machine learning

KW - Mathematical optimization

KW - Optimization with randomized search heuristics

KW - Theory of computation

UR - http://www.scopus.com/inward/record.url?scp=85134921867&partnerID=8YFLogxK

U2 - 10.1145/3520304.3533949

DO - 10.1145/3520304.3533949

M3 - Conference contribution

SP - 2113

EP - 2121

BT - Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022

CY - New York, NY, USA

T2 - 2022 Genetic and Evolutionary Computation Conference, GECCO 2022

Y2 - 9 July 2022 through 13 July 2022

ER -

Evolution of activation functions for deep learning-based image classification.

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this