共查询到20条相似文献，搜索用时 398 毫秒
1.
This study introduces a novel framework for evaluating passage and XML retrieval. The framework focuses on a user’s effort
to localize relevant content in a result document. Measuring the effort is based on a system guided reading order of documents.
The effort is calculated as the quantity of text the user is expected to browse through. More specifically, this study seeks
evaluation metrics for retrieval methods following a specific fetch and browse approach, where in the fetch phase documents
are ranked in decreasing order according to their document score, like in document retrieval. In the browse phase, for each
retrieved document, a set of non-overlapping passages representing the relevant text within the document is retrieved. In
other words, the passages of the document are re-organized, so that the best matching passages are read first in sequential
order. We introduce an application scenario motivating the framework, and propose sample metrics based on the framework. These
metrics give a basis for the comparison of effectiveness between traditional document retrieval and passage/XML retrieval
and illuminate the benefit of passage/XML retrieval. 相似文献
2.
Tove Faber Frandsen Christina Louise Lindhardt Mette Brandt Eriksen 《Journal of the Medical Library Association》2021,109(3):388
Objective:A growing volume of studies address methods for performing systematic reviews of qualitative studies. One such methodological aspect is the conceptual framework used to structure the review question and plan the search strategy for locating relevant studies. The purpose of this case study was to evaluate the retrieval potential of each element of conceptual frameworks in qualitative systematic reviews in the health sciences.Methods:The presence of elements from conceptual frameworks in publication titles, abstracts, and controlled vocabulary in CINAHL and PubMed was analyzed using a set of qualitative reviews and their included studies as a gold standard. Using a sample of 101 publications, we determined whether particular publications could be retrieved if a specific element from the conceptual framework was used in the search strategy.Results:We found that the relative recall of conceptual framework elements varied considerably, with higher recall for patient/population (99%) and research type (97%) and lower recall for intervention/phenomenon of interest (74%), outcome (79%), and context (61%).Conclusion:The use of patient/population and research type elements had high relative recall for qualitative studies. However, other elements should be used with great care due to lower relative recall. 相似文献
3.
Bevan Koopman Guido Zuccon Peter Bruza Laurianne Sitbon Michael Lawley 《Information Retrieval》2016,19(1-2):6-37
This paper presents a Graph Inference retrieval model that integrates structured knowledge resources, statistical information retrieval methods and inference in a unified framework. Key components of the model are a graph-based representation of the corpus and retrieval driven by an inference mechanism achieved as a traversal over the graph. The model is proposed to tackle the semantic gap problem—the mismatch between the raw data and the way a human being interprets it. We break down the semantic gap problem into five core issues, each requiring a specific type of inference in order to be overcome. Our model and evaluation is applied to the medical domain because search within this domain is particularly challenging and, as we show, often requires inference. In addition, this domain features both structured knowledge resources as well as unstructured text. Our evaluation shows that inference can be effective, retrieving many new relevant documents that are not retrieved by state-of-the-art information retrieval models. We show that many retrieved documents were not pooled by keyword-based search methods, prompting us to perform additional relevance assessment on these new documents. A third of the newly retrieved documents judged were found to be relevant. Our analysis provides a thorough understanding of when and how to apply inference for retrieval, including a categorisation of queries according to the effect of inference. The inference mechanism promoted recall by retrieving new relevant documents not found by previous keyword-based approaches. In addition, it promoted precision by an effective reranking of documents. When inference is used, performance gains can generally be expected on hard queries. However, inference should not be applied universally: for easy, unambiguous queries and queries with few relevant documents, inference did adversely affect effectiveness. These conclusions reflect the fact that for retrieval as inference to be effective, a careful balancing act is involved. Finally, although the Graph Inference model is developed and applied to medical search, it is a general retrieval model applicable to other areas such as web search, where an emerging research trend is to utilise structured knowledge resources for more effective semantic search. 相似文献
4.
Document clustering offers the potential of supporting users in interactive retrieval, especially when users have problems
in specifying their information need precisely. In this paper, we present a theoretic foundation for optimum document clustering.
Key idea is to base cluster analysis and evalutation on a set of queries, by defining documents as being similar if they are
relevant to the same queries. Three components are essential within our optimum clustering framework, OCF: (1) a set of queries,
(2) a probabilistic retrieval method, and (3) a document similarity metric. After introducing an appropriate validity measure,
we define optimum clustering with respect to the estimates of the relevance probability for the query-document pairs under
consideration. Moreover, we show that well-known clustering methods are implicitly based on the three components, but that
they use heuristic design decisions for some of them. We argue that with our framework more targeted research for developing
better document clustering methods becomes possible. Experimental results demonstrate the potential of our considerations. 相似文献
5.
In the field of scientometrics, impact indicators and ranking algorithms are frequently evaluated using unlabelled test data comprising relevant entities (e.g., papers, authors, or institutions) that are considered important. The rationale is that the higher some algorithm ranks these entities, the better its performance. To compute a performance score for an algorithm, an evaluation measure is required to translate the rank distribution of the relevant entities into a single-value performance score. Until recently, it was simply assumed that taking the average rank (of the relevant entities) is an appropriate evaluation measure when comparing ranking algorithms or fine-tuning algorithm parameters.With this paper we propose a framework for evaluating the evaluation measures themselves. Using this framework the following questions can now be answered: (1) which evaluation measure should be chosen for an experiment, and (2) given an evaluation measure and corresponding performance scores for the algorithms under investigation, how significant are the observed performance differences?Using two publication databases and four test data sets we demonstrate the functionality of the framework and analyse the stability and discriminative power of the most common information retrieval evaluation measures. We find that there is no clear winner and that the performance of the evaluation measures is highly dependent on the underlying data. Our results show that the average rank is indeed an adequate and stable measure. However, we also show that relatively large performance differences are required to confidently determine if one ranking algorithm is significantly superior to another. Lastly, we list alternative measures that also yield stable results and highlight measures that should not be used in this context. 相似文献
6.
To cope with the fact that, in the ad hoc retrieval setting, documents relevant to a query could contain very few (short)
parts (passages) with query-related information, researchers proposed passage-based document ranking approaches. We show that several of
these retrieval methods can be understood, and new ones can be derived, using the same probabilistic model. We use language-model
estimates to instantiate specific retrieval algorithms, and in doing so present a novel passage language model that integrates information from the containing document to an extent controlled by the estimated document homogeneity. Several document-homogeneity measures that we present yield passage language models that are more effective than the standard
passage model for basic document retrieval and for constructing and utilizing passage-based relevance models; these relevance models also outperform a document-based relevance model. Finally, we demonstrate the merits in using the
document-homogeneity measures for integrating document-query and passage-query similarity information for document retrieval. 相似文献
7.
Employing effective methods of sentence retrieval is essential for many tasks in Information Retrieval, such as summarization,
novelty detection and question answering. The best performing sentence retrieval techniques attempt to perform matching directly
between the sentences and the query. However, in this paper, we posit that the local context of a sentence can provide crucial
additional evidence to further improve sentence retrieval. Using a Language Modeling Framework, we propose a novel reformulation
of the sentence retrieval problem that extends previous approaches so that the local context is seamlessly incorporated within
the retrieval models. In a series of comprehensive experiments, we show that localized smoothing and the prior importance
of a sentence can improve retrieval effectiveness. The proposed models significantly and substantially outperform the state
of the art and other competitive sentence retrieval baselines on recall-oriented measures, while remaining competitive on
precision-oriented measures. This research demonstrates that local context plays an important role in estimating the relevance
of a sentence, and that existing sentence retrieval language models can be extended to utilize this evidence effectively. 相似文献
8.
Recently direct optimization of information retrieval (IR) measures has become a new trend in learning to rank. In this paper,
we propose a general framework for direct optimization of IR measures, which enjoys several theoretical advantages. The general
framework, which can be used to optimize most IR measures, addresses the task by approximating the IR measures and optimizing
the approximated surrogate functions. Theoretical analysis shows that a high approximation accuracy can be achieved by the
framework. We take average precision (AP) and normalized discounted cumulated gains (NDCG) as examples to demonstrate how
to realize the proposed framework. Experiments on benchmark datasets show that the algorithms deduced from our framework are
very effective when compared to existing methods. The empirical results also agree well with the theoretical results obtained
in the paper. 相似文献
9.
We propose a method for performing evaluation of relevance feedback based on simulating real users. The user simulation applies
a model defining the user’s relevance threshold to accept individual documents as feedback in a graded relevance environment;
user’s patience to browse the initial list of retrieved documents; and his/her effort in providing the feedback. We evaluate
the result by using cumulated gain-based evaluation together with freezing all documents seen by the user in order to simulate
the point of view of a user who is browsing the documents during the retrieval process. We demonstrate the method by performing
a simulation in the laboratory setting and present the “branching” curve sets characteristic for the presented evaluation
method. Both the average and topic-by-topic results indicate that if the freezing approach is adopted, giving feedback of
mixed quality makes sense for various usage scenarios even though the modeled users prefer finding especially the most relevant
documents. 相似文献
10.
David E. Losada 《Information Retrieval》2010,13(5):485-506
The retrieval of sentences that are relevant to a given information need is a challenging passage retrieval task. In this
context, the well-known vocabulary mismatch problem arises severely because of the fine granularity of the task. Short queries,
which are usually the rule rather than the exception, aggravate the problem. Consequently, effective sentence retrieval methods
tend to apply some form of query expansion, usually based on pseudo-relevance feedback. Nevertheless, there are no extensive
studies comparing different statistical expansion strategies for sentence retrieval. In this work we study thoroughly the
effect of distinct statistical expansion methods on sentence retrieval. We start from a set of retrieved documents in which
relevant sentences have to be found. In our experiments different term selection strategies are evaluated and we provide empirical
evidence to show that expansion before sentence retrieval yields competitive performance. This is particularly novel because expansion for sentence retrieval is often done after sentence retrieval (i.e. expansion terms are mined from a ranked set of sentences) and there are no comparative results available between both
types of expansion. Furthermore, this comparison is particularly valuable because there are important implications in time
efficiency. We also carefully analyze expansion on weak and strong queries and demonstrate clearly that expanding queries
before sentence retrieval is not only more convenient for efficiency purposes, but also more effective when handling poor
queries. 相似文献
11.
A General Evaluation Framework for Topical Crawlers 总被引：10，自引：0，他引：10
Topical crawlers are becoming important tools to support applications such as specialized Web portals, online searching, and competitive intelligence. As the Web mining field matures, the disparate crawling strategies proposed in the literature will have to be evaluated and compared on common tasks through well-defined performance measures. This paper presents a general framework to evaluate topical crawlers. We identify a class of tasks that model crawling applications of different nature and difficulty. We then introduce a set of performance measures for fair comparative evaluations of crawlers along several dimensions including generalized notions of precision, recall, and efficiency that are appropriate and practical for the Web. The framework relies on independent relevance judgements compiled by human editors and available from public directories. Two sources of evidence are proposed to assess crawled pages, capturing different relevance criteria. Finally we introduce a set of topic characterizations to analyze the variability in crawling effectiveness across topics. The proposed evaluation framework synthesizes a number of methodologies in the topical crawlers literature and many lessons learned from several studies conducted by our group. The general framework is described in detail and then illustrated in practice by a case study that evaluates four public crawling algorithms. We found that the proposed framework is effective at evaluating, comparing, differentiating and interpreting the performance of the four crawlers. For example, we found the IS crawler to be most sensitive to the popularity of topics.Partially supported by National Science Foundation CAREER grant No. IIS-0133124/0348940. 相似文献
12.
图书馆2.0个性化服务是图书馆服务的新理念和新方式,有必要从用户需求的角度对图书馆2.0个性化服务的应用模式进行研究。通过文献整理、问卷调查和案例分析,构建高校图书馆2.0个性化服务的分散模式和集中模式相结合的体系框架。分散模式包括:个性化定制、个性化推送、个性化信息互动、个性化参考咨询、个性化导航和个性化信息检索。在此基础上,构建出包括馆内外人员、信息资源、技术、服务、流程、平台和环境等各要素在内的高校图书馆2.0个性化服务的集中模式。 相似文献
13.
XML retrieval is a departure from standard document retrieval in which each individual XML element, ranging from italicized words or phrases to full blown articles, is a retrievable unit. The distribution of XML element lengths is unlike what we usually observe in standard document collections, prompting us to revisit the issue of document length normalization. We perform a comparative analysis of arbitrary elements versus relevant elements, and show the importance of element length as a parameter for XML retrieval. Within the language modeling framework, we investigate a range of techniques that deal with length either directly or indirectly. We observe a length-bias introduced by the amount of smoothing, and show the importance of extreme length bias for XML retrieval. We also show that simply removing shorter elements from the index (by introducing a cut-off value) does not create an appropriate element length normalization. Even after restricting the minimal size of XML elements occurring in the index, the importance of an extreme explicit length bias remains. 相似文献
14.
Robin Aly Aiden Doherty Djoerd Hiemstra Franciska de Jong Alan F. Smeaton 《Information Retrieval》2013,16(5):557-583
Concept based video retrieval often relies on imperfect and uncertain concept detectors. We propose a general ranking framework to define effective and robust ranking functions, through explicitly addressing detector uncertainty. It can cope with multiple concept-based representations per video segment and it allows the re-use of effective text retrieval functions which are defined on similar representations. The final ranking status value is a weighted combination of two components: the expected score of the possible scores, which represents the risk-neutral choice, and the scores’ standard deviation, which represents the risk or opportunity that the score for the actual representation is higher. The framework consistently improves the search performance in the shot retrieval task and the segment retrieval task over several baselines in five TRECVid collections and two collections which use simulated detectors of varying performance. 相似文献
15.
While past research has shown that learning outcomes can be influenced by the amount of effort students invest during the learning process, there has been little research into this question for scenarios where people use search engines to learn. In fact, learning-related tasks represent a significant fraction of the time users spend using Web search, so methods for evaluating and optimizing search engines to maximize learning are likely to have broad impact. Thus, we introduce and evaluate a retrieval algorithm designed to maximize educational utility for a vocabulary learning task, in which users learn a set of important keywords for a given topic by reading representative documents on diverse aspects of the topic. Using a crowdsourced pilot study, we compare the learning outcomes of users across four conditions corresponding to rankings that optimize for different levels of keyword density. We find that adding keyword density to the retrieval objective gave significant learning gains on some topics, with higher levels of keyword density generally corresponding to more time spent reading per word, and stronger learning gains per word read. We conclude that our approach to optimizing search ranking for educational utility leads to retrieved document sets that ultimately may result in more efficient learning of important concepts. 相似文献
16.
Investigating task performance of probabilistic topic models: an empirical study of PLSA and LDA 总被引：2，自引：0，他引：2
Probabilistic topic models have recently attracted much attention because of their successful applications in many text mining
tasks such as retrieval, summarization, categorization, and clustering. Although many existing studies have reported promising
performance of these topic models, none of the work has systematically investigated the task performance of topic models;
as a result, some critical questions that may affect the performance of all applications of topic models are mostly unanswered,
particularly how to choose between competing models, how multiple local maxima affect task performance, and how to set parameters
in topic models. In this paper, we address these questions by conducting a systematic investigation of two representative
probabilistic topic models, probabilistic latent semantic analysis (PLSA) and Latent Dirichlet Allocation (LDA), using three
representative text mining tasks, including document clustering, text categorization, and ad-hoc retrieval. The analysis of
our experimental results provides deeper understanding of topic models and many useful insights about how to optimize the
performance of topic models for these typical tasks. The task-based evaluation framework is generalizable to other topic models
in the family of either PLSA or LDA. 相似文献
17.
Anita A. H. Verhoeven Peter M. Boendermaker Edzard J. Boerma Betty Meyboom-De Jong 《Health information and libraries journal》1999,16(3):141-150
The aim of this study was to develop a model to evaluate the retrieval quality of search queries performed by Dutch general practitioners using the printed Index Medicus, MEDLINE on CD-ROM, and MEDLINE through GRATEFUL MED. Four search queries related to general practice were formulated for a continuing medical education course in literature searching. The selected potential relevant citations from the course instructor and the 103 course participants together served as the basic set for the three judges to evaluate for (a) relevance and (b) quality, with the latter based on journal ranking, research design and publication type. Relevant individual citations received a citation quality score from 1 (low) to 4 (high). The overall search quality was expressed in a formula, which included the individual citation quality score of the selected and missed relevant citations, and the number of selected non-relevant citations. The outcome measures were the number and quality of relevant citations and agreement between the judges. Out of 864 citations, 139 were assessed as relevant, of which 44 citations received an individual citation quality score of 1, 76 of 2, 19 of 3 and none of 4. The level of agreement between the judges was 68% for the relevant citations, and 88% for the non-relevant citations. We describe a model for the evaluation of search queries based not only on the relevance, but also on the quality of the citations retrieved. With adaptation, this model could be generalized to other professional users, and to other bibliographic sources. 相似文献
18.
19.
Stephen Robertson 《Information Retrieval》2002,5(2-3):239-256
An experimental adaptive filtering system, built on the Okapi search engine, is described. In addition to the regular text retrieval functions, the system requires a complex set of procedures for setting score thresholds and adapting them following feedback. These procedures need to be closely related to the evaluation measures to be used. A mixture of quantitative methods relating a threshold to the number of documents expected to be retrieved in a time period, and qualitative methods relating to the probability of relevance, is defined. Experiments under the TREC-9 Adaptive Filtering Track rules are reported. The system is seen to perform reasonably well in comparison with other systems at TREC. Some of the variables that may affect performance are investigated. 相似文献
20.
Content-oriented XML retrieval approaches aim at a more focused retrieval strategy: Instead of retrieving whole documents, document components that are exhaustive to the information need while at the same time being as specific as possible should be retrieved. In this article, we show that the evaluation methods developed for standard retrieval must be modified in order to deal with the structure of XML documents. More precisely, the size and overlap of document components must be taken into account. For this purpose, we propose a new effectiveness metric based on the definition of a concept space defined upon the notions of exhaustiveness and specificity of a search result. We compare the results of this new metric by the results obtained with the official metric used in INEX, the evaluation initiative for content-oriented XML retrieval.
相似文献
| Gabriella KazaiEmail: |