BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Elad Ben-Zaken; Shauli Ravfogel; Yoav Goldberg

doi:10.18653/v1/2022.acl-short.1

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg

Bar-Ilan University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

Original language	English
Title of host publication	ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)
Editors	Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Publisher	Association for Computational Linguistics (ACL)
Pages	1-9
Number of pages	9
ISBN (Electronic)	9781955917223
DOIs	https://doi.org/10.18653/v1/2022.acl-short.1
State	Published - 1 Jan 2022
Event	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland Duration: 22 May 2022 → 27 May 2022 https://aclanthology.org/2022.acl-long.0/

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
Volume	2

Conference

Conference	60th Annual Meeting of the Association for Computational Linguistics, ACL 2022
Country/Territory	Ireland
City	Dublin
Period	22/05/22 → 27/05/22
Internet address	https://aclanthology.org/2022.acl-long.0/

All Science Journal Classification (ASJC) codes

Computer Science Applications
Linguistics and Language
Language and Linguistics

Access to Document

10.18653/v1/2022.acl-short.1

Cite this

Ben-Zaken, E., Ravfogel, S., & Goldberg, Y. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers) (pp. 1-9). (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 2). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.acl-short.1

Ben-Zaken, Elad ; Ravfogel, Shauli ; Goldberg, Yoav. / BitFit : Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). editor / Smaranda Muresan ; Preslav Nakov ; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. pp. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{d509d8d6b505498b88a22b8f9d8d4e11,

title = "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models",

abstract = "We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.",

author = "Elad Ben-Zaken and Shauli Ravfogel and Yoav Goldberg",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 ; Conference date: 22-05-2022 Through 27-05-2022",

year = "2022",

month = jan,

day = "1",

doi = "10.18653/v1/2022.acl-short.1",

language = "الإنجليزيّة",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1--9",

editor = "Smaranda Muresan and Preslav Nakov and Aline Villavicencio",

booktitle = "ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)",

address = "الولايات المتّحدة",

url = "https://aclanthology.org/2022.acl-long.0/",

}

Ben-Zaken, E, Ravfogel, S & Goldberg, Y 2022, BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. in S Muresan, P Nakov & A Villavicencio (eds), ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 2, Association for Computational Linguistics (ACL), pp. 1-9, 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, Dublin, Ireland, 22/05/22. https://doi.org/10.18653/v1/2022.acl-short.1

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. / Ben-Zaken, Elad; Ravfogel, Shauli; Goldberg, Yoav.
ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). ed. / Smaranda Muresan; Preslav Nakov; Aline Villavicencio. Association for Computational Linguistics (ACL), 2022. p. 1-9 (Proceedings of the Annual Meeting of the Association for Computational Linguistics; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - BitFit

T2 - 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022

AU - Ben-Zaken, Elad

AU - Ravfogel, Shauli

AU - Goldberg, Yoav

PY - 2022/1/1

Y1 - 2022/1/1

N2 - We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

AB - We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.

UR - http://www.scopus.com/inward/record.url?scp=85142911444&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.acl-short.1

DO - 10.18653/v1/2022.acl-short.1

M3 - منشور من مؤتمر

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 1

EP - 9

BT - ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)

A2 - Muresan, Smaranda

A2 - Nakov, Preslav

A2 - Villavicencio, Aline

PB - Association for Computational Linguistics (ACL)

Y2 - 22 May 2022 through 27 May 2022

ER -

Ben-Zaken E, Ravfogel S, Goldberg Y. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In Muresan S, Nakov P, Villavicencio A, editors, ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers). Association for Computational Linguistics (ACL). 2022. p. 1-9. (Proceedings of the Annual Meeting of the Association for Computational Linguistics). doi: 10.18653/v1/2022.acl-short.1

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this