Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems |
| |
| Authors: | Phoebe M Roberts Aaron M Cohen William R Hersh |
| |
| Institution: | (1) Pfizer Research Technology Center, 620 Memorial Drive, Cambridge, MA 02139, USA;(2) Department of Medical Informatics and Clinical Epidemiology, School of Medicine, Oregon Health & Science University, 3181 S.W. Sam Jackson Park Road, Mail Code BICC, Portland, OR 97239-3098, USA |
| |
| Abstract: | With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of “gold standard”
test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding
relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and
training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific
backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about
factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods
used for identifying and training judges, and providing a central moderator “meta-judge”. |
| |
| Keywords: | Reference standards Evaluation Inter-annotator agreement Text mining Information retrieval |
| 本文献已被 SpringerLink 等数据库收录！ |
|
