Collaborative Inference via Ensembles on the Edge

Nir Shlezinger, Erez Farhan, Hai Morgenstern, Yonina C Eldar

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The success of deep neural networks (DNNs) as an enabler of artificial intelligence (AI) is heavily dependent on high computational resources. The increasing demands for accessible and personalized AI give rise to the need to operate DNNs on edge devices such as smartphones, sensors, and autonomous cars, whose computational powers are limited. Here we propose a framework for facilitating the application of DNNs on the edge in a manner which allows multiple users to collaborate during inference in order to improve their prediction accuracy. Our mechanism, referred to as edge ensembles, is based on having diverse predictors at each device, which can form a deep ensemble during inference. We analyze the latency induced in this collaborative inference approach, showing that the ability to improve performance via collaboration comes at the cost of a minor additional delay. Our experimental results demonstrate that collaborative inference via edge ensembles equipped with compact DNNs substantially improves the accuracy over having each user infer locally, and can outperform using a single centralized DNN larger than all the networks in the ensemble together.
Original languageEnglish
Title of host publicationICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Number of pages5
ISBN (Electronic)9781728176055
StatePublished - 13 May 2021
EventIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) - Toronto, ON, Canada
Duration: 6 Jun 202111 Jun 2021


ConferenceIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)


Dive into the research topics of 'Collaborative Inference via Ensembles on the Edge'. Together they form a unique fingerprint.

Cite this