共查询到20条相似文献，搜索用时 243 毫秒
1.
Simulation and analysis have shown that selective search can reduce the cost of large-scale distributed information retrieval. By partitioning the collection into small topical shards, and then using a resource ranking algorithm to choose a subset of shards to search for each query, fewer postings are evaluated. In this paper we extend the study of selective search into new areas using a fine-grained simulation, examining the difference in efficiency when term-based and sample-based resource selection algorithms are used; measuring the effect of two policies for assigning index shards to machines; and exploring the benefits of index-spreading and mirroring as the number of deployed machines is varied. Results obtained for two large datasets and four large query logs confirm that selective search is significantly more efficient than conventional distributed search architectures and can handle higher query rates. Furthermore, we demonstrate that selective search can be tuned to avoid bottlenecks, and thus maximize usage of the underlying computer hardware. 相似文献
2.
Suzan Verberne Maya Sappelli Djoerd Hiemstra Wessel Kraaij 《Information Retrieval》2016,19(5):510-545
We evaluate five term scoring methods for automatic term extraction on four different types of text collections: personal document collections, news articles, scientific articles and medical discharge summaries. Each collection has its own use case: author profiling, boolean query term suggestion, personalized query suggestion and patient query expansion. The methods for term scoring that have been proposed in the literature were designed with a specific goal in mind. However, it is as yet unclear how these methods perform on collections with characteristics different than what they were designed for, and which method is the most suitable for a given (new) collection. In a series of experiments, we evaluate, compare and analyse the output of six term scoring methods for the collections at hand. We found that the most important factors in the success of a term scoring method are the size of the collection and the importance of multi-word terms in the domain. Larger collections lead to better terms; all methods are hindered by small collection sizes (below 1000 words). The most flexible method for the extraction of single-word and multi-word terms is pointwise Kullback–Leibler divergence for informativeness and phraseness. Overall, we have shown that extracting relevant terms using unsupervised term scoring methods is possible in diverse use cases, and that the methods are applicable in more contexts than their original design purpose. 相似文献
3.
Query structuring and expansion with two-stage term dependence for Japanese web retrieval 总被引：1，自引：1，他引：0
In this paper, we propose a new term dependence model for information retrieval, which is based on a theoretical framework
using Markov random fields. We assume two types of dependencies of terms given in a query: (i) long-range dependencies that
may appear for instance within a passage or a sentence in a target document, and (ii) short-range dependencies that may appear
for instance within a compound word in a target document. Based on this assumption, our two-stage term dependence model captures
both long-range and short-range term dependencies differently, when more than one compound word appear in a query. We also
investigate how query structuring with term dependence can improve the performance of query expansion using a relevance model.
The relevance model is constructed using the retrieval results of the structured query with term dependence to expand the
query. We show that our term dependence model works well, particularly when using query structuring with compound words, through
experiments using a 100-gigabyte test collection of web documents mostly written in Japanese. We also show that the performance
of the relevance model can be significantly improved by using the structured query with our term dependence model.
相似文献
| Koji EguchiEmail: |
4.
James C. French Allison L. Powell Fredric Gey Natalia Perelman 《Information Retrieval》2002,5(4):323-351
Vocabulary incompatibilities arise when the terms used to index a document collection are largely unknown, or at least not well-known to the users who eventually search the collection. No matter how comprehensive or well-structured the indexing vocabulary, it is of little use if it is not used effectively in query formulation. This paper demonstrates that techniques for mapping user queries into the controlled indexing vocabulary have the potential to radically improve document retrieval performance. We also show how the use of controlled indexing vocabulary can be employed to achieve performance gains for collection selection. Finally, we demonstrate the potential benefit of combining these two techniques in an interactive retrieval environment. Given a user query, our evaluation approach simulates the human user's choice of terms for query augmentation given a list of controlled vocabulary terms suggested by a system. This strategy lets us evaluate interactive strategies without the need for human subjects. 相似文献
5.
LETOR: A benchmark collection for research on learning to rank for information retrieval 总被引：2，自引：0，他引：2
LETOR is a benchmark collection for the research on learning to rank for information retrieval, released by Microsoft Research
Asia. In this paper, we describe the details of the LETOR collection and show how it can be used in different kinds of researches.
Specifically, we describe how the document corpora and query sets in LETOR are selected, how the documents are sampled, how
the learning features and meta information are extracted, and how the datasets are partitioned for comprehensive evaluation.
We then compare several state-of-the-art learning to rank algorithms on LETOR, report their ranking performances, and make
discussions on the results. After that, we discuss possible new research topics that can be supported by LETOR, in addition
to algorithm comparison. We hope that this paper can help people to gain deeper understanding of LETOR, and enable more interesting
research projects on learning to rank and related topics. 相似文献
6.
一种支持高效检索的实时更新倒排索引策略 总被引：5，自引：0，他引：5
最近的研究使得搜索引擎中搜取的网页文档与万维网的变化越来越同步。为使用户通过搜索引擎获取网络上的最新信息,必须加快倒排索引的更新。本文介绍了使用界标和增加/删除网页文档两种典型的倒排索引更新策略,并分析了它们的优缺点,提出了一种支持高效检索的实时更新倒排索引策略。这种策略综合了减少更新操作、加快实时更新和缩短用户查询响应时间等方面的优点,较好地适应了当前网络内容变化的特点。最后通过实验对这种策略进行了验证。 相似文献
7.
Diomidis Spinellis 《Information Retrieval》2005,8(1):5-24
The infrastructure of a typical search engine can be used to calculate and resolve persistent document identifiers: a string that can uniquely identify and locate a document on the Internet without reference to its original location (URL). Bookmarking a document using such an identifier allows its retrieval even if the document's URL, and, in many cases, its contents change. Web client applications can offer facilities for users to bookmark a page by reference to a search engine and the persistent identifier instead of the original URL. The identifiers are calculated using a global Internet term index; a document's unique identifier consists of a word or word combination that occurs uniquely in the specific document. We use a genetic algorithm to locate a minimal unique document identifier: the shortest word or word combination that will locate the document. We tested our approach by implementing tools for indexing a document collection, calculating the persistent identifiers, performing queries, and distributing the computation and storage load among many computers. 相似文献
8.
In many probabilistic modeling approaches to Information Retrieval we are interested in estimating how well a document model
“fits” the user’s information need (query model). On the other hand in statistics, goodness of fit tests are well established
techniques for assessing the assumptions about the underlying distribution of a data set. Supposing that the query terms are
randomly distributed in the various documents of the collection, we actually want to know whether the occurrences of the query
terms are more frequently distributed by chance in a particular document. This can be quantified by the so-called goodness
of fit tests. In this paper, we present a new document ranking technique based on Chi-square goodness of fit tests. Given
the null hypothesis that there is no association between the query terms q and the document d irrespective of any chance occurrences, we perform a Chi-square goodness of fit test for assessing this hypothesis and calculate
the corresponding Chi-square values. Our retrieval formula is based on ranking the documents in the collection according to
these calculated Chi-square values. The method was evaluated over the entire test collection of TREC data, on disks 4 and
5, using the topics of TREC-7 and TREC-8 (50 topics each) conferences. It performs well, outperforming steadily the classical
OKAPI term frequency weighting formula but below that of KL-Divergence from language modeling approach. Despite this, we believe
that the technique is an important non-parametric way of thinking of retrieval, offering the possibility to try simple alternative
retrieval formulas within goodness-of-fit statistical tests’ framework, modeling the data in various ways estimating or assigning any arbitrary theoretical distribution
in terms. 相似文献
9.
10.
Index maintenance strategies employed by dynamic text retrieval systems based on inverted files can be divided into two categories:
merge-based and in-place update strategies. Within each category, individual update policies can be distinguished based on
whether they store their on-disk posting lists in a contiguous or in a discontiguous fashion. Contiguous inverted lists, in
general, lead to higher query performance, by minimizing the disk seek overhead at query time, while discontiguous inverted
lists lead to higher update performance, requiring less effort during index maintenance operations. In this paper, we focus
on retrieval systems with high query load, where the on-disk posting lists have to be stored in a contiguous fashion at all
times. We discuss a combination of re-merge and in-place index update, called Hybrid Immediate Merge. The method performs strictly better than the re-merge baseline policy used in our experiments, as it leads to the same query
performance, but substantially better update performance. The actual time savings achievable depend on the size of the text
collection being indexed; a larger collection results in greater savings. In our experiments, variations of Hybrid Immediate Merge were able to reduce the total index update overhead by up to 73% compared to the re-merge baseline.
相似文献
| Stefan BüttcherEmail: |
11.
The explosion of content in distributed information retrieval (IR) systems requires new mechanisms in order to attain timely and accurate retrieval of unstructured text. This paper shows how to exploit locality by building, using, and searching partial replicas of text collections in a distributed IR system. In this work, a partial replica includes a subset of the documents from larger collection(s) and the corresponding inference network search mechanism. For each query, the distributed system determines if partial replica is a good match and then searches it, or it searches the original collection. We demonstrate the scenarios where partial replication performs better than systems that use caches which only store previous query and answer pairs. We first use logs from THOMAS and Excite to examine query locality using query similarity versus exact match. We show that searching replicas can improve locality (from 3 to 19%) over the exact match required by caching. Replicas increase locality because they satisfy queries which are distinct but return the same or very similar answers. We then present a novel inference network replica selection function. We vary its parameters and compare it to previous collection selection functions, demonstrating a configuration that directs most of the appropriate queries to replicas in a replica hierarchy. We then explore the performance of partial replication in a distributed IR system. We compare it with caching and partitioning. Our validated simulator shows that the increases in locality due to replication make it preferable to caching alone, and that even a small increase of 4% in locality translates into a performance advantage. We also show a hybrid system with caches and replicas that performs better than each on their own. 相似文献
12.
Multilingual retrieval (querying of multiple document collections each in a different language) can be achieved by combining several individual techniques which enhance retrieval: machine translation to cross the language barrier, relevance feedback to add words to the initial query, decompounding for languages with complex term structure, and data fusion to combine monolingual retrieval results from different languages. Using the CLEF 2001 and CLEF 2002 topics and document collections, this paper evaluates these techniques within the context of a monolingual document ranking formula based upon logistic regression. Each individual technique yields improved performance over runs which do not utilize that technique. Moreover the techniques are complementary, in that combining the best techniques outperforms individual technique performance. An approximate but fast document translation using bilingual wordlists created from machine translation systems is presented and evaluated. The fast document translation is as effective as query translation in multilingual retrieval. Furthermore, when fast document translation is combined with query translation in multilingual retrieval, the performance is significantly better than that of query translation or fast document translation. 相似文献
13.
In Information Retrieval, since it is hard to identify users’ information needs, many approaches have been tried to solve
this problem by expanding initial queries and reweighting the terms in the expanded queries using users’ relevance judgments.
Although relevance feedback is most effective when relevance information about retrieved documents is provided by users, it
is not always available. Another solution is to use correlated terms for query expansion. The main problem with this approach
is how to construct the term-term correlations that can be used effectively to improve retrieval performance. In this study,
we try to construct query concepts that denote users’ information needs from a document space, rather than to reformulate initial queries using the term correlations
and/or users’ relevance feedback. To form query concepts, we extract features from each document, and then cluster the features into primitive concepts that are then used to form
query concepts. Experiments are performed on the Associated Press (AP) dataset taken from the TREC collection. The experimental evaluation
shows that our proposed framework called QCM (Query Concept Method) outperforms baseline probabilistic retrieval model on
TREC retrieval. 相似文献
14.
考察在信息检索过程中用户以自然语言表述的查询语句中的词语使用情况。以一个信息需求描述颗粒度不等的查询表述语料库为素材,辅以汉语通用语料作为对照,通过词频以及词语的文本覆盖率等统计数据,按照是否需要在目标文本中直接或以其他形式出现,将查询表述语句中的词语区分为对汉语文本处理具有普遍意义的通用停用词、服务于信息检索表述用的专用停用词和与特定需求相关的信息内容词语。区分词语使用的不同性质,能为信息系统前端的自然语言查询处理增加一道剥离工序,防止将整个查询语句的分词结果全部作为检索项所造成的效率和准确率的退化。 相似文献
15.
检索词自动扩展词库构建方法的基本思路是：根据语料是否规范化处理进行词库分类建设,优化了系统的检索性能;结合学科类别,对词库语料进行领域划分,引导科技人员对技术领域的准确把握;建设以本体库为基础,将与规范词具有关联性、相似性的语料通过关系表与关联库关联,把科技文献中的关键词组成一个有序的关系网,解决了传统检索系统中检索词无关联的不足;通过对检索词出现频率进行统计分析,进而更新词库,保证本体库、关联库语料的时效性,突破了人工对词库更新管理的受限性。 相似文献
16.
We consider a multi-stage retrieval architecture consisting of a fast, “cheap” candidate generation stage, a feature extraction stage, and a more “expensive” reranking stage using machine-learned models. In this context, feature extraction can be accomplished using a document vector index, a mapping from document ids to document representations. We consider alternative organizations of such a data structure for efficient feature extraction: design choices include how document terms are organized, how complex term proximity features are computed, and how these structures are compressed. In particular, we propose a novel document-adaptive hashing scheme for compactly encoding term ids. The impact of alternative designs on both feature extraction speed and memory footprint is experimentally evaluated. Overall, results show that our architecture is comparable in speed to using a traditional positional inverted index but requires less memory overall, and offers additional advantages in terms of flexibility. 相似文献
17.
We consider the following autocompletion search scenario: imagine a user of a search engine typing a query; then with every
keystroke display those completions of the last query word that would lead to the best hits, and also display the best such
hits. The following problem is at the core of this feature: for a fixed document collection, given a set D of documents, and an alphabetical range W of words, compute the set of all word-in-document pairs (w, d) from the collection such that w ∈ W and d ∈ D. We present a new data structure with the help of which such autocompletion queries can be processed, on the average, in
time linear in the input plus output size, independent of the size of the underlying document collection. At the same time,
our data structure uses no more space than an inverted index. Actual query processing times on a large test collection correlate
almost perfectly with our theoretical bound.
相似文献
| Ingmar WeberEmail: |
18.
TIJAH: Embracing IR Methods in XML Databases 总被引：1，自引：0，他引：1
This paper discusses our participation in INEX (the Initiative for the Evaluation of XML Retrieval) using the TIJAH XML-IR system. TIJAHs system design follows a standard layered database architecture, carefully separating the conceptual, logical and physical levels. At the conceptual level, we classify the INEX XPath-based query expressions into three different query patterns. For each pattern, we present its mapping into a query execution strategy. The logical layer exploits score region algebra (SRA) as the basis for query processing. We discuss the region operators used to select and manipulate XML document components. The logical algebra expressions are mapped into efficient relational algebra expressions over a physical representation of the XML document collection using the pre-post numbering scheme. The paper concludes with an analysis of experiments performed with the INEX test collection. 相似文献
19.
This paper presents four novel techniques for open-vocabulary spoken document retrieval: a method to detect slots that possibly contain a query feature; a method to estimate occurrence probabilities; a technique that we call collection-wide probability re-estimation and a weighting scheme which takes advantage of the fact that long query features are detected more reliably. These four techniques have been evaluated using the TREC-6 spoken document retrieval test collection to determine the improvements in retrieval effectiveness with respect to a baseline retrieval method. Results show that the retrieval effectiveness can be improved considerably despite the large number of speech recognition errors. 相似文献
20.
Recent studies demonstrated that it is possible to reduce Inverted Files (IF) sizes by reassigning the document identifiers of the original collection, as this lowers the distance between the positions
of documents related to a single term. Variable-bit encoding schemes can exploit the average gap reduction and decrease the
total amount of bits per document pointer. This paper presents an efficient solution to the reassignment problem, which consists
in reducing the input data dimensionality using a SVD transformation, as well as considering it a Travelling Salesman Problem (TSP). We also present some efficient solutions based on clustering. Finally, we combine both the TSP and the clustering
strategies for reordering the document identifiers. We present experimental tests and performance results in two text TREC
collections, obtaining good compression ratios with low running times, and advance the possibility of obtaining scalable solutions
for web collections based on the techniques presented here. 相似文献