An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

Marco Cognetta; Tatsuya Hiraoka; Naoaki Okazaki; Rico Sennrich; Yuval Pinter

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

Marco Cognetta, Tatsuya Hiraoka, Naoaki Okazaki, Rico Sennrich, Yuval Pinter

Department of Computer Science

Ben-Gurion University of the Negev

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a tokenization postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in model implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to consistently improve model performance, and is even prone to incurring heavy degradation.

Original language	American English
Title of host publication	Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop
Editors	Shabnam Tafreshi, Arjun Reddy Akula, Joao Sedoc, Aleksandr Drozd, Anna Rogers, Anna Rumshisky
Publisher	Association for Computational Linguistics (ACL)
Pages	48-50
Number of pages	3
ISBN (Electronic)	9798891761025
State	Published - 1 Jan 2024
Event	5th Workshop on Insights from Negative Results in NLP, Insights 2024 - Mexico City, Mexico Duration: 20 Jun 2024 → …

Publication series

Name	Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop

Conference

Conference	5th Workshop on Insights from Negative Results in NLP, Insights 2024
Country/Territory	Mexico
City	Mexico City
Period	20/06/24 → …

All Science Journal Classification (ASJC) codes

Language and Linguistics
Computational Theory and Mathematics
Computer Science Applications
Software

Cite this

Cognetta, M., Hiraoka, T., Okazaki, N., Sennrich, R., & Pinter, Y. (2024). An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. In S. Tafreshi, A. R. Akula, J. Sedoc, A. Drozd, A. Rogers, & A. Rumshisky (Eds.), Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop (pp. 48-50). (Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

Cognetta, Marco ; Hiraoka, Tatsuya ; Okazaki, Naoaki et al. / An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop. editor / Shabnam Tafreshi ; Arjun Reddy Akula ; Joao Sedoc ; Aleksandr Drozd ; Anna Rogers ; Anna Rumshisky. Association for Computational Linguistics (ACL), 2024. pp. 48-50 (Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop).

@inproceedings{be4ff54f7f154bb99b86a0de3155c762,

title = "An Analysis of BPE Vocabulary Trimming in Neural Machine Translation",

abstract = "We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a tokenization postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in model implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to consistently improve model performance, and is even prone to incurring heavy degradation.",

author = "Marco Cognetta and Tatsuya Hiraoka and Naoaki Okazaki and Rico Sennrich and Yuval Pinter",

note = "Publisher Copyright: {\textcopyright} 2024 Association for Computational Linguistics.; 5th Workshop on Insights from Negative Results in NLP, Insights 2024 ; Conference date: 20-06-2024",

year = "2024",

month = jan,

day = "1",

language = "American English",

series = "Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "48--50",

editor = "Shabnam Tafreshi and Akula, {Arjun Reddy} and Joao Sedoc and Aleksandr Drozd and Anna Rogers and Anna Rumshisky",

booktitle = "Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop",

address = "United States",

}

Cognetta, M, Hiraoka, T, Okazaki, N, Sennrich, R & Pinter, Y 2024, An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. in S Tafreshi, AR Akula, J Sedoc, A Drozd, A Rogers & A Rumshisky (eds), Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop. Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 48-50, 5th Workshop on Insights from Negative Results in NLP, Insights 2024, Mexico City, Mexico, 20/06/24.

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. / Cognetta, Marco; Hiraoka, Tatsuya; Okazaki, Naoaki et al.
Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop. ed. / Shabnam Tafreshi; Arjun Reddy Akula; Joao Sedoc; Aleksandr Drozd; Anna Rogers; Anna Rumshisky. Association for Computational Linguistics (ACL), 2024. p. 48-50 (Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

AU - Cognetta, Marco

AU - Hiraoka, Tatsuya

AU - Okazaki, Naoaki

AU - Sennrich, Rico

AU - Pinter, Yuval

PY - 2024/1/1

Y1 - 2024/1/1

N2 - We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a tokenization postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in model implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to consistently improve model performance, and is even prone to incurring heavy degradation.

AB - We explore threshold vocabulary trimming in Byte-Pair Encoding subword tokenization, a tokenization postprocessing step that replaces rare subwords with their component subwords. The technique is available in popular tokenization libraries but has not been subjected to rigorous scientific scrutiny. While the removal of rare subwords is suggested as best practice in model implementations, both as a means to reduce model size and for improving model performance through robustness, our experiments indicate that, across a large space of hyperparameter settings, vocabulary trimming fails to consistently improve model performance, and is even prone to incurring heavy degradation.

UR - http://www.scopus.com/inward/record.url?scp=105000186400&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop

SP - 48

EP - 50

BT - Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop

A2 - Tafreshi, Shabnam

A2 - Akula, Arjun Reddy

A2 - Sedoc, Joao

A2 - Drozd, Aleksandr

A2 - Rogers, Anna

A2 - Rumshisky, Anna

PB - Association for Computational Linguistics (ACL)

T2 - 5th Workshop on Insights from Negative Results in NLP, Insights 2024

Y2 - 20 June 2024

ER -

Cognetta M, Hiraoka T, Okazaki N, Sennrich R, Pinter Y. An Analysis of BPE Vocabulary Trimming in Neural Machine Translation. In Tafreshi S, Akula AR, Sedoc J, Drozd A, Rogers A, Rumshisky A, editors, Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2024. p. 48-50. (Insights 2024 - 5th Workshop on Insights from Negative Results in NLP, Proceedings of the Workshop).

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this