Show simple item record

dc.contributor.authorKanojia, Diptesh
dc.contributor.authorFomicheva, Marina
dc.contributor.authorRanasinghe, Tharindu
dc.contributor.authorBlain, Frederic
dc.contributor.authorOrasan, Constantin
dc.contributor.authorSpecia, Lucia
dc.contributor.editorBarrault, Loricen
dc.contributor.editorBojar, Ondrejen
dc.contributor.editorBougaris, Fethien
dc.contributor.editorChatterjee, Rajenen
dc.contributor.editorCosta-jussa, Marta R.en
dc.contributor.editorFedermann, Christianen
dc.contributor.editorFishel, Marken
dc.contributor.editorFraser, Alexanderen
dc.contributor.editorFreitag, Markusen
dc.contributor.editorGraham, Yvetteen
dc.contributor.editorGrundkiewicz, Romanen
dc.contributor.editorGuzman, Pacoen
dc.contributor.editorHaddow, Barryen
dc.contributor.editorHuck, Matthiasen
dc.contributor.editorYepes, Antonio Jimenoen
dc.contributor.editorKoehn, Philippen
dc.contributor.editorKocmi, Tomen
dc.contributor.editorMartins, Andreen
dc.contributor.editorMorishita, Makotoen
dc.contributor.editorMonz, Christofen
dc.date.accessioned2021-09-29T08:15:22Z
dc.date.available2021-09-29T08:15:22Z
dc.date.issued2022-01-11
dc.identifier.citationKanojia, D., Fomicheva, M., Ranasinghe, T., Blain, F., Orasan, C. and Specia, L. (2021) Pushing the Right Buttons: Adversarial Evaluation of Quality Estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 625–638, Online. Association for Computational Linguistics.en
dc.identifier.isbn9781954085947
dc.identifier.urihttp://hdl.handle.net/2436/624376
dc.description© (2021) The Authors. Published by Association for Computational Linguistics. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://aclanthology.org/2021.wmt-1.67en
dc.description.abstractCurrent Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaningpreserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.en
dc.formatapplication/pdfen
dc.language.isoenen
dc.publisherAssociation for Computational Linguisticsen
dc.relation.urlhttps://aclanthology.org/2021.wmt-1.67en
dc.subjectadversarial evaluationen
dc.subjectmachine translationen
dc.subjectquality estimationen
dc.titlePushing the right buttons: adversarial evaluation of quality estimationen
dc.typeConference contributionen
dc.date.updated2021-09-27T15:14:46Z
dc.conference.nameEMNLP 2021 Sixth Conference on Machine Translation (WMT21)
dc.conference.locationOnline
pubs.finish-date2021-11-11
pubs.start-date2021-11-10
dc.date.accepted2021-09-07
rioxxterms.funderUniversity of Wolverhamptonen
rioxxterms.identifier.projectUOW29092021FBen
rioxxterms.versionVoRen
rioxxterms.licenseref.urihttps://creativecommons.org/licenses/by/4.0/en
rioxxterms.licenseref.startdate2022-01-14en
dc.source.booktitleProceedings of the Sixth Conference on Machine Translationen
dc.source.beginpage625
dc.source.endpage638
refterms.dateFCD2021-09-29T08:14:49Z
refterms.versionFCDVoR
refterms.dateFOA2022-01-14T11:28:59Z


Files in this item

Thumbnail
Name:
2109.10859.pdf
Size:
822.0Kb
Format:
PDF

This item appears in the following Collection(s)

Show simple item record

https://creativecommons.org/licenses/by/4.0/
Except where otherwise noted, this item's license is described as https://creativecommons.org/licenses/by/4.0/