Pooling for a Large-Scale Test Collection: An Analysis of the Search Results from the First NTCIR Workshop |
| |
| Authors: | Kazuko Kuriyama Noriko Kando Toshihiko Nozue Koji Eguchi |
| |
| Institution: | (1) National Institutes of Informatics (NII), 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan;(2) Aoyama Gakuin University, 4-4-25 Shibuya, Shibuya-ku, Tokyo, 150-8366, Japan;(3) National Institutes of Informatics (NII), 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan |
| |
| Abstract: | We have conducted a study to: (1) verify the exhaustiveness of pooling for the purpose of constructing a large-scale test collection, and (2) examine whether a difference in the number of pool documents can affect the relative evaluation of IR systems. We carried out the experiments using search topics, their relevance assessments, and the search results that were submitted for both the pre-test and test of the first NTCIR Workshop.Our results verified the efficiency and the effectiveness of the pooling method, the exhaustiveness of the relevance assessments, and the reliability of the evaluation using the test collection based on the pooling method. |
| |
| Keywords: | test collection IR system evaluation methodology relevance assessment exhaustiveness of pooling |
| 本文献已被 SpringerLink 等数据库收录！ |
|
