Event

Baselines and Limits in Extractive Meeting Summarization

Korbinian Riedhammer

Monday, July 07, 2008
12:30

After a short introduction to summarization, I'll describe two meeting data sets (AMI and ICSI) and their annotations. Current state of the art automatic evaluation methods include the text summarization rooted ROUGE and a weighted precision measure. In preparation for understanding the limits in extractive summarization, I'll give detailed examples for these measures. The important reason for baseline and limit results is that prior works on meeting summarization always changed preprocessing, summary lengths and evaluation criterion, which makes it very hard to compare algorithms and results. Accompanying new results with baseline and limit results for the same conditions allows a comparison between algorithms and results. To do so, I introduce two simple baselines for summarization (random selection and longest utterances). To determine the upper limit, we mapped the summarization problem to a knapsack problem, searching for the best subset of utterances to achieve the best evaluation score while satisfying a given length constraint. We solve that optimization problem with a linear integer program and give results for manual transcripts and ASR data. Finally, I give a brief outlook on further work to do in meeting summarization.