Using Mechanical Turk to Build Machine Translation Evaluation Sets

View/ Open
Date
2010-06Author
Bloodgood, Michael
Callison-Burch, Chris
Citation
Michael Bloodgood and Chris Callison-Burch. 2010. Using mechanical turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 208-211, Los Angeles, California, June. Association for Computational Linguistics.
Metadata
Show full item recordAbstract
Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon’s Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professionally-produced test sets. More importantly, in experiments with multiple MT systems, we find that the MTurk-produced test sets yield essentially the same conclusions regarding system performance as the professionally-produced test sets yield.
Related items
Showing items related by title, author, creator and subject.
-
Searching to Translate and Translating to Search: When Information Retrieval Meets Machine Translation
Ture, Ferhan (2013)With the adoption of web services in daily life, people have access to tremendous amounts of information, beyond any human's reading and comprehension capabilities. As a result, search technologies have become a fundamental ... -
Handling Translation Divergences in Generation-Heavy Hybrid Machine Translation
Habash, Nizar; Dorr, Bonnie (2002-04-04)This paper describes a novel approach for handling translation divergences in a Generation-Heavy Hybrid Machine Translation (GHMT) system. The approach depends on the existence of rich target language resources such as ... -
Handling Translation Divergences: Combining Statistical and Symbolic Techniques in Generation-Heavy Machine Translation
Habash, Nizar; Dorr, Bonnie (2002-05-22)This paper describes a novel approach to handling translation divergences in a Generation-Heavy Hybrid Machine Translation (GHMT) system.The translation divergence problem is usually reserved for Transfer and Interlingual ...