Loading...
Pushing the right buttons: adversarial evaluation of quality estimation
Kanojia, Diptesh Fomicheva, Marina Ranasinghe, Tharindu Blain, Frederic Orasan, Constantin Specia, Lucia
Kanojia, Diptesh
Fomicheva, Marina
Ranasinghe, Tharindu
Blain, Frederic
Orasan, Constantin
Specia, Lucia
Editors
Barrault, Loric
Bojar, Ondrej
Bougaris, Fethi
Chatterjee, Rajen
Costa-jussa, Marta R.
Federmann, Christian
Fishel, Mark
Fraser, Alexander
Freitag, Markus
Graham, Yvette
Grundkiewicz, Roman
Guzman, Paco
Haddow, Barry
Huck, Matthias
Yepes, Antonio Jimeno
Koehn, Philipp
Kocmi, Tom
Martins, Andre
Morishita, Makoto
Monz, Christof
Bojar, Ondrej
Bougaris, Fethi
Chatterjee, Rajen
Costa-jussa, Marta R.
Federmann, Christian
Fishel, Mark
Fraser, Alexander
Freitag, Markus
Graham, Yvette
Grundkiewicz, Roman
Guzman, Paco
Haddow, Barry
Huck, Matthias
Yepes, Antonio Jimeno
Koehn, Philipp
Kocmi, Tom
Martins, Andre
Morishita, Makoto
Monz, Christof
Other contributors
Affiliation
Epub Date
Issue Date
2022-01-11
Submitted date
Alternative
Abstract
Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaningpreserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.
Citation
Kanojia, D., Fomicheva, M., Ranasinghe, T., Blain, F., Orasan, C. and Specia, L. (2021) Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 625–638, Online. Association for Computational Linguistics.
Journal
Research Unit
DOI
PubMed ID
PubMed Central ID
Embedded videos
Additional Links
Type
Conference contribution
Language
en
Description
© (2021) The Authors. Published by Association for Computational Linguistics. This is an open access article available under a Creative Commons licence.
The published version can be accessed at the following link on the publisher’s website: https://aclanthology.org/2021.wmt-1.67
Series/Report no.
ISSN
EISSN
ISBN
9781954085947