Skip to content
University of Maryland LibrariesDigital Repository at the University of Maryland
    • Login
    View Item 
    •   DRUM
    • Center for Advanced Study of Language
    • Center for Advanced Study of Language Research Works
    • View Item
    •   DRUM
    • Center for Advanced Study of Language
    • Center for Advanced Study of Language Research Works
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Using Mechanical Turk to Build Machine Translation Evaluation Sets

    Thumbnail
    View/Open
    crowdsourcingMachineTranslationEvaluationNAACLWorkshop2010.pdf (95.38Kb)
    No. of downloads: 211

    Date
    2010-06
    Author
    Bloodgood, Michael
    Callison-Burch, Chris
    Citation
    Michael Bloodgood and Chris Callison-Burch. 2010. Using mechanical turk to build machine translation evaluation sets. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 208-211, Los Angeles, California, June. Association for Computational Linguistics.
    Metadata
    Show full item record
    Abstract
    Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon’s Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professionally-produced test sets. More importantly, in experiments with multiple MT systems, we find that the MTurk-produced test sets yield essentially the same conclusions regarding system performance as the professionally-produced test sets yield.
    URI
    http://hdl.handle.net/1903/15551
    Collections
    • Center for Advanced Study of Language Research Works

    Related items

    Showing items related by title, author, creator and subject.

    • Searching to Translate and Translating to Search: When Information Retrieval Meets Machine Translation 

      Ture, Ferhan (2013)
      With the adoption of web services in daily life, people have access to tremendous amounts of information, beyond any human's reading and comprehension capabilities. As a result, search technologies have become a fundamental ...
    • Handling Translation Divergences in Generation-Heavy Hybrid Machine Translation 

      Habash, Nizar; Dorr, Bonnie (2002-04-04)
      This paper describes a novel approach for handling translation divergences in a Generation-Heavy Hybrid Machine Translation (GHMT) system. The approach depends on the existence of rich target language resources such as ...
    • Handling Translation Divergences: Combining Statistical and Symbolic Techniques in Generation-Heavy Machine Translation 

      Habash, Nizar; Dorr, Bonnie (2002-05-22)
      This paper describes a novel approach to handling translation divergences in a Generation-Heavy Hybrid Machine Translation (GHMT) system.The translation divergence problem is usually reserved for Transfer and Interlingual ...

    DRUM is brought to you by the University of Maryland Libraries
    University of Maryland, College Park, MD 20742-7011 (301)314-1328.
    Please send us your comments.
    Web Accessibility
     

     

    Browse

    All of DRUMCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

    My Account

    LoginRegister
    Pages
    About DRUMAbout Download Statistics

    DRUM is brought to you by the University of Maryland Libraries
    University of Maryland, College Park, MD 20742-7011 (301)314-1328.
    Please send us your comments.
    Web Accessibility