Mathematics
Permanent URI for this communityhttp://hdl.handle.net/1903/2261
Browse
2 results
Search Results
Item Better Metrics to Automatically Predict the Quality of a Text Summary(MDPI, 2012-09-26) Rankel, Peter A.; Conroy, John M.; Schlesinger, Judith D.In this paper we demonstrate a family of metrics for estimating the quality of a text summary relative to one or more human-generated summaries. The improved metrics are based on features automatically computed from the summaries to measure content and linguistic quality. The features are combined using one of three methods—robust regression, non-negative least squares, or canonical correlation, an eigenvalue method. The new metrics significantly outperform the previous standard for automatic text summarization evaluation, ROUGE.Item Statistical Analysis of Text Summarization Evaluation(2016) Rankel, Peter A.; Slud, Eric V.; Conroy, John M.; Mathematical Statistics; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)This dissertation applies statistical methods to the evaluation of automatic summarization using data from the Text Analysis Conferences in 2008-2011. Several aspects of the evaluation framework itself are studied, including the statistical testing used to determine significant differences, the assessors, and the design of the experiment. In addition, a family of evaluation metrics is developed to predict the score an automatically generated summary would receive from a human judge and its results are demonstrated at the Text Analysis Conference. Finally, variations on the evaluation framework are studied and their relative merits considered. An over-arching theme of this dissertation is the application of standard statistical methods to data that does not conform to the usual testing assumptions.