I really enjoy "serendipitous browsing" or "exploratory search" on the Web. You start off with a specific goal and you end up reading all sorts of papers and articles on the Web! And again while I was looking for a specific paper at the ACM DL, I ended up reading the ACM transactions on Speech and Language Processing (TSLP) and found this article: "The Pyramid Method: Incorporating Human Content Selection Variation in Summarisation Evaluation". I think summarisation can be useful for visually and cognitively disabled users. Although there has been some work on automating the summarisation process, it is very difficult to evaluate the quality of automatically generated summaries. This paper talks about the human variation in content summarisation and focuses on the difficulty of evaluating automatically generated summaries. Basically, one can easily create an algorithm to automatically generate summaries, but what about the quality of these generated summaries? This paper proposes a method that can be used to create multiple human models, which can then be used as a gold-standard for evaluation. The Pyramid Method: Incorporating Human Content Selection Variation in Summarisation Evaluation

Human variation in content selection in summarization has given rise to some fundamental research questions: How can one incorporate the observed variation in suitable evaluation measures? How can such measures reflect the fact that summaries conveying different content can be equally good and informative? In this article, we address these very questions by proposing a method for analysis of multiple human abstracts into semantic content units. Such analysis allows us not only to quantify human variation in content selection, but also to assign empirical importance weight to different content units. It serves as the basis for an evaluation method, the Pyramid Method, that incorporates the observed variation and is predictive of different equally informative summaries. We discuss the reliability of content unit annotation, the properties of Pyramid scores, and their correlation with other evaluation methods.

Full Paper: http://doi.acm.org/10.1145/1233912.1233913 Full Proceedings: ACM transactions on Speech and Language Processing (TSLP), Vol. 4, Number 2, Publication date: 2007

Labels: browsing, exploratory search, summarisation