Test Collection Management and Labeling System Conference Paper uri icon

abstract

  • In order to evaluate the performance of information retrieval and extraction algorithms, we need test collections. A test collection consists of a set of documents, a clearly formed problem that an algorithm is supposed to provide solutions to, and the answers that the algorithm should produce when executed on the documents. Defining the association between elements in the test collection and answers is known as labeling. For mainstream information retrieval problems, there are publicly available test collections which have been maintained for years. However, the scope of these problems, and thus the associated test collections, is limited. In other cases, researchers need to build, label, and manage their own test collections, which can be a tedious and error-prone task. We built test collections of HTML documents, for problems in which the answer that the algorithm supplies is a sub-tree of the DOM (Document Object Model). To lighten the burden of this task, we developed a test collection management and labeling system (TCMLS), to facilitate usability in the process of building test collections, applying them to validate algorithms, and potentially sharing them across the research community. Copyright 2009 ACM.

name of conference

  • Proceedings of the 9th ACM symposium on Document engineering

published proceedings

  • DOCENG'09: PROCEEDINGS OF THE 2009 ACM SYMPOSIUM ON DOCUMENT ENGINEERING

author list (cited authors)

  • Koh, E., Kerne, A., & Berry, S.

citation count

  • 2

complete list of authors

  • Koh, Eunyee||Kerne, Andruid||Berry, Sarah

editor list (cited editors)

  • Borghoff, U. M., & Chidlovskii, B.

publication date

  • January 2009