TRANSLATOTRON 3: SPEECH TO SPEECH TRANSLATION WITH MONOLINGUAL DATA

Eliya Nachmani; Alon Levkovitch; Yifan Ding; Chulayuth Asawaroengchai; Heiga Zen; Michelle Tadmor Ramanovich

doi:10.1109/ICASSP48485.2024.10448426

TRANSLATOTRON 3: SPEECH TO SPEECH TRANSLATION WITH MONOLINGUAL DATA

Eliya Nachmani
, Alon Levkovitch
, Yifan Ding
, Chulayuth Asawaroengchai
, Heiga Zen
, Michelle Tadmor Ramanovich

COMPUTER SCIENCE

Tel Aviv University

Research output: Contribution to journal › Conference article › peer-review

Abstract

This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation. Experimental results in speech-to-speech translation tasks between Spanish and English show that Translatotron 3 outperforms a baseline cascade system, reporting 18.14 BLEU points improvement on the synthesized Unpaired-Conversational dataset. In contrast to supervised approaches that necessitate real paired data, or specialized modeling to replicate para-/non-linguistic information such as pauses, speaking rates, and speaker identity, Translatotron 3 showcases its capability to retain it.

Original language	English
Pages (from-to)	10686-10690
Number of pages	5
Journal	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
DOIs	https://doi.org/10.1109/ICASSP48485.2024.10448426
State	Published - 1 Jan 2024
Event	2024 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2024 - Seoul, Korea, Republic of Duration: 14 Apr 2024 → 19 Apr 2024

Keywords

Speech-to-speech translation
Unsupervised

ASJC Scopus subject areas

Software
Signal Processing
Electrical and Electronic Engineering

Access to Document

10.1109/ICASSP48485.2024.10448426

Cite this

@article{87da5bb5143c434d83f263bcc6f971ab,

title = "TRANSLATOTRON 3: SPEECH TO SPEECH TRANSLATION WITH MONOLINGUAL DATA",

abstract = "This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation. Experimental results in speech-to-speech translation tasks between Spanish and English show that Translatotron 3 outperforms a baseline cascade system, reporting 18.14 BLEU points improvement on the synthesized Unpaired-Conversational dataset. In contrast to supervised approaches that necessitate real paired data, or specialized modeling to replicate para-/non-linguistic information such as pauses, speaking rates, and speaker identity, Translatotron 3 showcases its capability to retain it.",

keywords = "Speech-to-speech translation, Unsupervised",

author = "Eliya Nachmani and Alon Levkovitch and Yifan Ding and Chulayuth Asawaroengchai and Heiga Zen and Ramanovich, \{Michelle Tadmor\}",

note = "Publisher Copyright: {\textcopyright}2024 IEEE.; 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2024 ; Conference date: 14-04-2024 Through 19-04-2024",

year = "2024",

month = jan,

day = "1",

doi = "10.1109/ICASSP48485.2024.10448426",

language = "English",

pages = "10686--10690",

journal = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

issn = "1520-6149",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - TRANSLATOTRON 3

T2 - 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2024

AU - Nachmani, Eliya

AU - Levkovitch, Alon

AU - Ding, Yifan

AU - Asawaroengchai, Chulayuth

AU - Zen, Heiga

AU - Ramanovich, Michelle Tadmor

PY - 2024/1/1

Y1 - 2024/1/1

N2 - This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation. Experimental results in speech-to-speech translation tasks between Spanish and English show that Translatotron 3 outperforms a baseline cascade system, reporting 18.14 BLEU points improvement on the synthesized Unpaired-Conversational dataset. In contrast to supervised approaches that necessitate real paired data, or specialized modeling to replicate para-/non-linguistic information such as pauses, speaking rates, and speaker identity, Translatotron 3 showcases its capability to retain it.

AB - This paper presents Translatotron 3, a novel approach to unsupervised direct speech-to-speech translation from monolingual speech-text datasets by combining masked autoencoder, unsupervised embedding mapping, and back-translation. Experimental results in speech-to-speech translation tasks between Spanish and English show that Translatotron 3 outperforms a baseline cascade system, reporting 18.14 BLEU points improvement on the synthesized Unpaired-Conversational dataset. In contrast to supervised approaches that necessitate real paired data, or specialized modeling to replicate para-/non-linguistic information such as pauses, speaking rates, and speaker identity, Translatotron 3 showcases its capability to retain it.

KW - Speech-to-speech translation

KW - Unsupervised

UR - https://www.scopus.com/pages/publications/105001483286

U2 - 10.1109/ICASSP48485.2024.10448426

DO - 10.1109/ICASSP48485.2024.10448426

M3 - Conference article

SN - 1520-6149

SP - 10686

EP - 10690

JO - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

JF - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Y2 - 14 April 2024 through 19 April 2024

ER -

TRANSLATOTRON 3: SPEECH TO SPEECH TRANSLATION WITH MONOLINGUAL DATA

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this