Findings of the first shared task on machine translation robustness

Xian Li, Paul Michel, Antonios Anastasopoulos, Yonatan Belinkov, Nadir Durrani, Orhan Firat, Philipp Koehn, Graham Neubig, Juan Pino, Hassan Sajjad

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

We share the findings of the first shared task on improving robustness of Machine Translation (MT). The task provides a testbed representing challenges facing MT models deployed in the real world, and facilitates new approaches to improve models' robustness to noisy input and domain mismatch. We focus on two language pairs (English-French and English-Japanese), and the submitted systems are evaluated on a blind test set consisting of noisy comments on Reddit and professionally sourced translations. As a new task, we received 23 submissions by 11 participating teams from universities, companies, national labs, etc. All submitted systems achieved large improvements over baselines, with the best improvement having +22.33 BLEU. We evaluated submissions by both human judgment and automatic evaluation (BLEU), which shows high correlations (Pearson's r = 0.94 and 0.95). Furthermore, we conducted a qualitative analysis of the submitted systems using compare-mt, which revealed their salient differences in handling challenges in this task. Such analysis provides additional insights when there is occasional disagreement between human judgment and BLEU, e.g. systems better at producing colloquial expressions received higher score from human judgment.

Original languageEnglish
Title of host publicationShared Task Papers, Day 1
Pages91-102
Number of pages12
ISBN (Electronic)9781950737277
StatePublished - 2019
Externally publishedYes
Event4th Conference on Machine Translation, WMT 2019 held at the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019 - Florence, Italy
Duration: 1 Aug 20192 Aug 2019

Publication series

NameWMT 2019 - 4th Conference on Machine Translation, Proceedings of the Conference
Volume2

Conference

Conference4th Conference on Machine Translation, WMT 2019 held at the 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019
Country/TerritoryItaly
CityFlorence
Period1/08/192/08/19

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Human-Computer Interaction
  • Software

Fingerprint

Dive into the research topics of 'Findings of the first shared task on machine translation robustness'. Together they form a unique fingerprint.

Cite this