EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning

Shay Vargaftik; Ran Ben Basat; Amit Portnoy; Gal Mendelson; Yaniv Ben Itzhak; Michael Mitzenmacher

EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning

Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben Itzhak, Michael Mitzenmacher

Data and Decision Sciences

Technion - Israel Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this work, we propose a robust DME technique named EDEN that naturally handles heterogeneous communication budgets and packet losses. We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques.

Original language	American English
Title of host publication	Proceedings of the 39th International Conference on Machine Learning
Editors	Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, Sivan Sabato
Pages	21984-22014
Number of pages	31
Volume	162
State	Published - 1 May 2022

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR

Access to Document

https://proceedings.mlr.press/v162/vargaftik22a.html

Cite this

Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G., Itzhak, Y. B., & Mitzenmacher, M. (2022). EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning (Vol. 162, pp. 21984-22014). (Proceedings of Machine Learning Research). https://proceedings.mlr.press/v162/vargaftik22a.html

Vargaftik, Shay ; Basat, Ran Ben ; Portnoy, Amit et al. / EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning. Proceedings of the 39th International Conference on Machine Learning. editor / Kamalika Chaudhuri ; Stefanie Jegelka ; Le Song ; Csaba Szepesvari ; Gang Niu ; Sivan Sabato. Vol. 162 2022. pp. 21984-22014 (Proceedings of Machine Learning Research).

@inproceedings{d9dc515563ed47c19da9a1e84c0c285a,

title = "EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning",

abstract = "Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this work, we propose a robust DME technique named EDEN that naturally handles heterogeneous communication budgets and packet losses. We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques.",

author = "Shay Vargaftik and Basat, {Ran Ben} and Amit Portnoy and Gal Mendelson and Itzhak, {Yaniv Ben} and Michael Mitzenmacher",

year = "2022",

month = may,

day = "1",

language = "American English",

volume = "162",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "21984--22014",

editor = "Kamalika Chaudhuri and Stefanie Jegelka and Le Song and Csaba Szepesvari and Gang Niu and Sivan Sabato",

booktitle = "Proceedings of the 39th International Conference on Machine Learning",

}

Vargaftik, S, Basat, RB, Portnoy, A, Mendelson, G, Itzhak, YB & Mitzenmacher, M 2022, EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning. in K Chaudhuri, S Jegelka, L Song, C Szepesvari, G Niu & S Sabato (eds), Proceedings of the 39th International Conference on Machine Learning. vol. 162, Proceedings of Machine Learning Research, pp. 21984-22014. <https://proceedings.mlr.press/v162/vargaftik22a.html>

EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning. / Vargaftik, Shay; Basat, Ran Ben; Portnoy, Amit et al.
Proceedings of the 39th International Conference on Machine Learning. ed. / Kamalika Chaudhuri; Stefanie Jegelka; Le Song; Csaba Szepesvari; Gang Niu; Sivan Sabato. Vol. 162 2022. p. 21984-22014 (Proceedings of Machine Learning Research).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning

AU - Vargaftik, Shay

AU - Basat, Ran Ben

AU - Portnoy, Amit

AU - Mendelson, Gal

AU - Itzhak, Yaniv Ben

AU - Mitzenmacher, Michael

PY - 2022/5/1

Y1 - 2022/5/1

N2 - Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this work, we propose a robust DME technique named EDEN that naturally handles heterogeneous communication budgets and packet losses. We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques.

AB - Distributed Mean Estimation (DME) is a central building block in federated learning, where clients send local gradients to a parameter server for averaging and updating the model. Due to communication constraints, clients often use lossy compression techniques to compress the gradients, resulting in estimation inaccuracies. DME is more challenging when clients have diverse network conditions, such as constrained communication budgets and packet losses. In such settings, DME techniques often incur a significant increase in the estimation error leading to degraded learning performance. In this work, we propose a robust DME technique named EDEN that naturally handles heterogeneous communication budgets and packet losses. We derive appealing theoretical guarantees for EDEN and evaluate it empirically. Our results demonstrate that EDEN consistently improves over state-of-the-art DME techniques.

M3 - Conference contribution

VL - 162

T3 - Proceedings of Machine Learning Research

SP - 21984

EP - 22014

BT - Proceedings of the 39th International Conference on Machine Learning

A2 - Chaudhuri, Kamalika

A2 - Jegelka, Stefanie

A2 - Song, Le

A2 - Szepesvari, Csaba

A2 - Niu, Gang

A2 - Sabato, Sivan

ER -

Vargaftik S, Basat RB, Portnoy A, Mendelson G, Itzhak YB, Mitzenmacher M. EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learning. In Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G, Sabato S, editors, Proceedings of the 39th International Conference on Machine Learning. Vol. 162. 2022. p. 21984-22014. (Proceedings of Machine Learning Research).