Center for Advanced Study of Language Research Works

Permanent URI for this collection

http://hdl.handle.net/1903/11610

Browse

Browsing Center for Advanced Study of Language Research Works by Subject "agreement metrics"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Analysis of Stopping Active Learning based on Stabilizing Predictions
    (Association for Computational Linguistics, 2013-08) Bloodgood, Michael; Grothendieck, John
    Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stopping active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the success of the SP method: (1) bounds on Cohen’s Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be labeled, it can be made large in practice, helping to guarantee that the results transfer to previously unseen streams of examples at test/application time; and (3) good (low variance) sample estimates of Kappa between successive models can be obtained. Proofs of relationships between the level of Kappa agreement and the difference in performance between consecutive models are presented. Specifically, if the Kappa agreement between two models exceeds a threshold T (where T > 0), then the difference in F-measure performance between those models is bounded above by 4(1−T)/T in all cases. If precision of the positive conjunction of the models is assumed to be p, then the bound can be tightened to 4(1−T)/((p+1)T).
  • Thumbnail Image
    Item
    A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping
    (Association for Computational Linguistics, 2009-06) Bloodgood, Michael; Vijay-Shanker, K
    A survey of existing methods for stopping active learning (AL) reveals the needs for methods that are: more widely applicable; more aggressive in saving annotations; and more stable across changing datasets. A new method for stopping AL based on stabilizing predictions is presented that addresses these needs. Furthermore, stopping methods are required to handle a broad range of different annotation/performance tradeoff valuations. Despite this, the existing body of work is dominated by conservative methods with little (if any) attention paid to providing users with control over the behavior of stopping methods. The proposed method is shown to fill a gap in the level of aggressiveness available for stopping AL and supports providing users with control over stopping behavior.