Breaking Character: Are Subwords Good Enough for MRLs After All?

Omri Keren, Tal Avinari, Reut Tsarfaty, Omer Levy

Research output: Working paperPreprint

Abstract

Large pretrained language models (PLMs) typically tokenize the input string into contiguous subwords before any pretraining or inference. However, previous studies have claimed that this form of subword tokenization is inadequate for processing morphologically-rich languages (MRLs). We revisit this hypothesis by pretraining a BERT-style masked language model over character sequences instead of word-pieces. We compare the resulting model, dubbed TavBERT, against contemporary PLMs based on subwords for three highly complex and ambiguous MRLs (Hebrew, Turkish, and Arabic), testing them on both morphological and semantic tasks. Our results show, for all tested languages, that while TavBERT obtains mild improvements on surface-level tasks à la POS tagging and full morphological disambiguation, subword-based PLMs achieve significantly higher performance on semantic tasks, such as named entity recognition and extractive question answering. These results showcase and (re)confirm the potential of subword tokenization as a reasonable modeling assumption for many languages, including MRLs.
Original languageAmerican English
Volumeabs/2204.04748
DOIs
StatePublished - 2022

Publication series

NameCoRR
PublisherCornell University Library, arXiv.org

Fingerprint

Dive into the research topics of 'Breaking Character: Are Subwords Good Enough for MRLs After All?'. Together they form a unique fingerprint.

Cite this