Skip to content
University of Maryland LibrariesDigital Repository at the University of Maryland
    • Login
    View Item 
    •   DRUM
    • Center for Advanced Study of Language
    • Center for Advanced Study of Language Research Works
    • View Item
    •   DRUM
    • Center for Advanced Study of Language
    • Center for Advanced Study of Language Research Works
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Data Cleaning for XML Electronic Dictionaries via Statistical Anomaly Detection

    Thumbnail
    View/Open
    dataCleaningICSC2016.pdf (324.5Kb)
    No. of downloads: 326

    Date
    2016
    Author
    Bloodgood, Michael
    Strauss, Benjamin
    Citation
    Michael Bloodgood and Benjamin Strauss. Data cleaning for XML electronic dictionaries via statistical anomaly detection. In Proceedings of the 2016 IEEE Tenth International Conference on Semantic Computing (ICSC), pages 79-86, Laguna Hills, CA, USA, February 2016. IEEE.
    DRUM DOI
    https://doi.org/10.13016/M2RT7D
    Metadata
    Show full item record
    Abstract
    Many important forms of data are stored digitally in XML format. Errors can occur in the textual content of the data in the fields of the XML. Fixing these errors manually is time-consuming and expensive, especially for large amounts of data. There is increasing interest in the research, development, and use of automated techniques for assisting with data cleaning. Electronic dictionaries are an important form of data frequently stored in XML format that frequently have errors introduced through a mixture of manual typographical entry errors and optical character recognition errors. In this paper we describe methods for flagging statistical anomalies as likely errors in electronic dictionaries stored in XML format. We describe six systems based on different sources of information. The systems detect errors using various signals in the data including uncommon characters, text length, character-based language models, word-based language models, tied-field length ratios, and tied-field transliteration models. Four of the systems detect errors based on expectations automatically inferred from content within elements of a single field type. We call these single-field systems. Two of the systems detect errors based on correspondence expectations automatically inferred from content within elements of multiple related field types. We call these tied-field systems. For each system, we provide an intuitive analysis of the type of error that it is successful at detecting. Finally, we describe two larger-scale evaluations using crowdsourcing with Amazon’s Mechanical Turk platform and using the annotations of a domain expert. The evaluations consistently show that the systems are useful for improving the efficiency with which errors in XML electronic dictionaries can be detected.
    URI
    http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7439308
    http://hdl.handle.net/1903/17459
    Collections
    • Center for Advanced Study of Language Research Works

    DRUM is brought to you by the University of Maryland Libraries
    University of Maryland, College Park, MD 20742-7011 (301)314-1328.
    Please send us your comments.
    Web Accessibility
     

     

    Browse

    All of DRUMCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

    My Account

    LoginRegister
    Pages
    About DRUMAbout Download Statistics

    DRUM is brought to you by the University of Maryland Libraries
    University of Maryland, College Park, MD 20742-7011 (301)314-1328.
    Please send us your comments.
    Web Accessibility