Breaking Character: Are Subwords Good Enough for MRLs After All?

Omri Keren, Tal Avinari, Reut Tsarfaty, Omer Levy

Research output: Working paperPreprint


Large pretrained language models (PLMs) typically tokenize the input string into contiguous subwords before any pretraining or inference. However, previous studies have claimed that this form of subword tokenization is inadequate for processing morphologically-rich languages (MRLs). We revisit this hypothesis by pretraining a BERT-style masked language model over character sequences instead of word-pieces. We compare the resulting model, dubbed TavBERT, against contemporary PLMs based on subwords for three highly complex and ambiguous MRLs (Hebrew, Turkish, and Arabic), testing them on both morphological and semantic tasks. Our results show, for all tested languages, that while TavBERT obtains mild improvements on surface-level tasks à la POS tagging and full morphological disambiguation, subword-based PLMs achieve significantly higher performance on semantic tasks, such as named entity recognition and extractive question answering. These results showcase and (re)confirm the potential of subword tokenization as a reasonable modeling assumption for many languages, including MRLs.
Original languageAmerican English
StatePublished - 2022

Publication series

PublisherCornell University Library,


Dive into the research topics of 'Breaking Character: Are Subwords Good Enough for MRLs After All?'. Together they form a unique fingerprint.

Cite this