共查询到20条相似文献，搜索用时 46 毫秒
1.
Significant progress has been made in information retrieval covering text semantic indexing and multilingual analysis. However,
developments in Arabic information retrieval did not follow the extraordinary growth of Arabic usage in the Web during the
ten last years. In the tasks relating to semantic analysis, it is preferable to directly deal with texts in their original
language. Studies on topic models, which provide a good way to automatically deal with semantic embedded in texts, are not
complete enough to assess the effectiveness of the approach on Arabic texts. This paper investigates several text stemming
methods for Arabic topic modeling. A new lemma-based stemmer is described and applied to newspaper articles. The Latent Dirichlet
Allocation model is used to extract latent topics from three Arabic real-world corpora. For supervised classification in the
topics space, experiments show an improvement when comparing to classification in the full words space or with root-based
stemming approach. In addition, topic modeling with lemma-based stemming allows us to discover interesting subjects in the
press articles published during the 2007–2009 period. 相似文献
2.
James C. French Allison L. Powell Fredric Gey Natalia Perelman 《Information Retrieval》2002,5(4):323-351
Vocabulary incompatibilities arise when the terms used to index a document collection are largely unknown, or at least not well-known to the users who eventually search the collection. No matter how comprehensive or well-structured the indexing vocabulary, it is of little use if it is not used effectively in query formulation. This paper demonstrates that techniques for mapping user queries into the controlled indexing vocabulary have the potential to radically improve document retrieval performance. We also show how the use of controlled indexing vocabulary can be employed to achieve performance gains for collection selection. Finally, we demonstrate the potential benefit of combining these two techniques in an interactive retrieval environment. Given a user query, our evaluation approach simulates the human user's choice of terms for query augmentation given a list of controlled vocabulary terms suggested by a system. This strategy lets us evaluate interactive strategies without the need for human subjects. 相似文献
3.
Towards enhancing retrieval effectiveness of search engines for diacritisized Arabic documents 总被引：1，自引：1，他引：0
Bassam H. Hammo 《Information Retrieval》2009,12(3):300-323
4.
Fernando Martínez-Santiago L. Alfonso Ureña-López Maite Martín-Valdivia 《Information Retrieval》2006,9(1):71-93
A usual strategy to implement CLIR (Cross-Language Information Retrieval) systems is the so-called query translation approach.
The user query is translated for each language present in the multilingual collection in order to compute an independent monolingual
information retrieval process per language. Thus, this approach divides documents according to language. In this way, we obtain
as many different collections as languages. After searching in these corpora and obtaining a result list per language, we
must merge them in order to provide a single list of retrieved articles.
In this paper, we propose an approach to obtain a single list of relevant documents for CLIR systems driven by query translation.
This approach, which we call 2-step RSV (RSV: Retrieval Status Value), is based on the re-indexing of the retrieval documents
according to the query vocabulary, and it performs noticeably better than traditional methods.
The proposed method requires query vocabulary alignment: given a word for a given query, we must know the translation or translations
to the other languages. Because this is not always possible, we have researched on a mixed model. This mixed model is applied
in order to deal with queries with partial word-level alignment. The results prove that even in this scenario, 2-step RSV
performs better than traditional merging methods. 相似文献
5.
Recent years have witnessed considerable advances in information retrieval for European languages other than English. We give an overview of commonly used techniques and we analyze them with respect to their impact on retrieval effectiveness. The techniques considered range from linguistically motivated techniques, such as morphological normalization and compound splitting, to knowledge-free approaches, such as n-gram indexing. Evaluations are carried out against data from the CLEF campaign, covering eight European languages. Our results show that for many of these languages a modicum of linguistic techniques may lead to improvements in retrieval effectiveness, as can the use of language independent techniques. 相似文献
6.
Automatic detection of source code plagiarism is an important research field for both the commercial software industry and within the research community. Existing methods of plagiarism detection primarily involve exhaustive pairwise document comparison, which does not scale well for large software collections. To achieve scalability, we approach the problem from an information retrieval (IR) perspective. We retrieve a ranked list of candidate documents in response to a pseudo-query representation constructed from each source code document in the collection. The challenge in source code document retrieval is that the standard bag-of-words (BoW) representation model for such documents is likely to result in many false positives being retrieved, because of the use of identical programming language specific constructs and keywords. To address this problem, we make use of an abstract syntax tree (AST) representation of the source code documents. While the IR approach is efficient, it is essentially unsupervised in nature. To further improve its effectiveness, we apply a supervised classifier (pre-trained with features extracted from sample plagiarized source code pairs) on the top ranked retrieved documents. We report experiments on the SOCO-2014 dataset comprising 12K Java source files with almost 1M lines of code. Our experiments confirm that the AST based approach produces significantly better retrieval effectiveness than a standard BoW representation, i.e., the AST based approach is able to identify a higher number of plagiarized source code documents at top ranks in response to a query source code document. The supervised classifier, trained on features extracted from sample plagiarized source code pairs, is shown to effectively filter and thus further improve the ranked list of retrieved candidate plagiarized documents. 相似文献
7.
User generated content forms an important domain for mining knowledge. In this paper, we address the task of blog feed search:
to find blogs that are principally devoted to a given topic, as opposed to blogs that merely happen to mention the topic in
passing. The large number of blogs makes the blogosphere a challenging domain, both in terms of effectiveness and of storage
and retrieval efficiency. We examine the effectiveness of an approach to blog feed search that is based on individual posts
as indexing units (instead of full blogs). Working in the setting of a probabilistic language modeling approach to information
retrieval, we model the blog feed search task by aggregating over a blogger’s posts to collect evidence of relevance to the
topic and persistence of interest in the topic. This approach achieves state-of-the-art performance in terms of effectiveness.
We then introduce a two-stage model where a pre-selection of candidate blogs is followed by a ranking step. The model integrates
aggressive pruning techniques as well as very lean representations of the contents of blog posts, resulting in substantial
gains in efficiency while maintaining effectiveness at a very competitive level. 相似文献
8.
Athena Salaba 《Cataloging & classification quarterly》2013,51(1):23-51
This study examines end-user interactions with indexing language information during subject searching in a library catalog and their understanding of this information and its function in term selection. Participants were asked to interact with the indexing language (Library of Congress Subject Headings) and were asked to express their general understanding of the information provided and each specific type of information included in the indexing language. In addition, participants were asked to express their understanding of the function of indexing language information in term selection, its usefulness and desirability as an integrated tool into the information retrieval system during subject searching. Study findings and their implications are discussed and future research is considered. 相似文献
9.
An information retrieval (IR) system can often fail to retrieve relevant documents due to the incomplete specification of information need in the user’s query. Pseudo-relevance feedback (PRF) aims to improve IR effectiveness by exploiting potentially relevant aspects of the information need present in the documents retrieved in an initial search. Standard PRF approaches utilize the information contained in these top ranked documents from the initial search with the assumption that documents as a whole are relevant to the information need. However, in practice, documents are often multi-topical where only a portion of the documents may be relevant to the query. In this situation, exploitation of the topical composition of the top ranked documents, estimated with statistical topic modeling based approaches, can potentially be a useful cue to improve PRF effectiveness. The key idea behind our PRF method is to use the term-topic and the document-topic distributions obtained from topic modeling over the set of top ranked documents to re-rank the initially retrieved documents. The objective is to improve the ranks of documents that are primarily composed of the relevant topics expressed in the information need of the query. Our RF model can further be improved by making use of non-parametric topic modeling, where the number of topics can grow according to the document contents, thus giving the RF model the capability to adjust the number of topics based on the content of the top ranked documents. We empirically validate our topic model based RF approach on two document collections of diverse length and topical composition characteristics: (1) ad-hoc retrieval using the TREC 6-8 and the TREC Robust ’04 dataset, and (2) tweet retrieval using the TREC Microblog ’11 dataset. Results indicate that our proposed approach increases MAP by up to 9% in comparison to the results obtained with an LDA based language model (for initial retrieval) coupled with the relevance model (for feedback). Moreover, the non-parametric version of our proposed approach is shown to be more effective than its parametric counterpart due to its advantage of adapting the number of topics, improving results by up to 5.6% of MAP compared to the parametric version. 相似文献
10.
We present a novel approach to re-ranking a document list that was retrieved in response to a query so as to improve precision
at the very top ranks. The approach is based on utilizing a second list that was retrieved in response to the query by using,
for example, a different retrieval method and/or query representation. In contrast to commonly-used methods for fusion of retrieved lists that rely solely on retrieval scores (ranks) of documents, our approach also exploits inter-document-similarities between the lists—a potentially rich source of additional information. Empirical evaluation shows that our methods are effective
in re-ranking TREC runs; the resultant performance also favorably compares with that of a highly effective fusion method.
Furthermore, we show that our methods can potentially help to tackle a long-standing challenge, namely, integration of document-based
and cluster-based retrieved results. 相似文献
11.
基于概念空间方法的信息检索技术研究 总被引：14，自引：0，他引：14
为了解决词汇差异问题,词表构造在信息检索系统中有着重要意义。概念空间方法是利用计算机自动构造概念语义网络(词表)并以此为基础进行概念检索的一种方法。由词语作为语义网络的节点,词语之间的关联权重以一个给定文档集合中词语的共现率来计算,其大小代表它们之间的相似性。检索时系统采用人工智能方法激活与检索入口词相关的术语或概念,为用户提供交互式的检索用语建议。方法的具体步骤包括文档和对象列表收集、对象过滤和自动标引、共现分析和联想检索四个阶段。这种方法多用于英文检索系统,但对我国的信息检索系统也有重要的借鉴意义。 相似文献
12.
语言模型在信息检索中的应用 总被引：1，自引：0，他引：1
基于语言模型的检索方法为信息检索领域开辟了一个很有前景同时也具有相当挑战性的方向。与传统检索模型相比,语言模型不仅具有良好的理论基础,而且非常灵活,经过简单的变换很容易推演出其他经典的检索模型。此外,大量的实验结果表明,该方法的检索效果优于其他检索模型,因而一经提出便受到了广大研究人员的青睐。然而当前语言模型方法的研究主要集中在单语检索任务中,很少有研究关注语言模型方法在跨语言检索中的应用,针对这个问题,本文在系统介绍基于语言模型检索方法的基础上,将语言模型方法扩展到跨语言检索任务中,介绍了两个跨语言检索模型:统计翻译模型和跨语言相关语言模型。 相似文献
13.
Jacques Savoy 《Information Retrieval》2004,7(1-2):121-148
This paper describes and evaluates different retrieval strategies that are useful for search operations on document collections written in various European languages, namely French, Italian, Spanish and German. We also suggest and evaluate different query translation schemes based on freely available translation resources. In order to cross language barriers, we propose a combined query translation approach that has resulted in interesting retrieval effectiveness. Finally, we suggest a collection merging strategy based on logistic regression that tends to perform better than other merging approaches. 相似文献
14.
对开源全文检索引擎Lucene的系统架构、索引与检索过程、语言分析器进行分析的基础上,针对其对中文只能进行单字切分、双字切分的不足,二次开发基于Lucene中英文语言分析器ZH_CNAnalyzer,并给出一个调用此分析器建立索引与检索的实例。 相似文献
15.
Improving English and Chinese Ad-Hoc Retrieval: A Tipster Text Phase 3 Project Report 总被引：1，自引：1，他引：0
K.L. Kwok 《Information Retrieval》2000,3(4):313-338
Both English and Chinese ad-hoc information retrieval were investigated in this Tipster 3 project. Part of our objectives is to study the use of various term level and phrasal level evidence to improve retrieval accuracy. For short queries, we studied five term level techniques that together can lead to good improvements over standard ad-hoc 2-stage retrieval for TREC5-8 experiments. For long queries, we studied the use of linguistic phrases to re-rank retrieval lists. Its effect is small but consistently positive.For Chinese IR, we investigated three simple representations for documents and queries: short-words, bigrams and characters. Both approximate short-word segmentation or bigrams, augmented with characters, give highly effective results. Accurate word segmentation appears not crucial for overall result of a query set. Character indexing by itself is not competitive. Additional improvements may be obtained using collection enrichment and combination of retrieval lists.Our PIRCS document-focused retrieval is also shown to have similarity with a simple language model approach to IR. 相似文献
16.
Christoph Baumgarten 《Information Retrieval》2000,3(3):253-271
This paper describes a probabilistic model for optimum information retrieval in a distributed heterogeneous environment.The model assumes the collection of documents offered by the environment to be partitioned into subcollections. Documents as well as subcollections have to be indexed, where indexing methods using different indexing vocabularies can be employed. A query provided by a user is answered in terms of a ranked list of documents. The model determines a procedure for ranking the documents that stems from the Probability Ranking Principle: For each subcollection, the subcollection's documents are ranked; the resulting ranked lists are combined into a final ranked list of documents, where the ordering is determined by the documents' probabilities of being relevant with respect to the user's query. Various probabilistic ranking methods may be involved in the distributed ranking process. A criterion for effectively limiting the ranking process to a subset of subcollections extends the model.The property that different ranking methods and indexing vocabularies can be used is important when the subcollections are heterogeneous with respect to their content.The model's applicability is experimentally confirmed. When exploiting the degrees of freedom provided by the model, experiments showed evidence that the model even outperforms comparable models for the non-distributed case with respect to retrieval effectiveness. 相似文献
17.
18.
In this paper, which treats Swedish full text retrieval, the problem of morphological variation of query terms in the document database is studied. The Swedish CLEF 2003 test collection was used, and the effects of combination of indexing strategies with query terms on retrieval effectiveness were studied. Four of the seven tested combinations involved indexing strategies that used normalization, a form of conflation. All of these four combinations employed compound splitting, both during indexing and at query phase. SWETWOL, a morphological analyzer for the Swedish language, was used for normalization and compound splitting. A fifth combination used stemming, while a sixth attempted to group related terms by right hand truncation of query terms. The truncation was performed by a search expert. These six combinations were compared to each other and to a baseline combination, where no attempt was made to counteract the problem of morphological variation of query terms in the document database. Both the truncation combination, the four combinations based on normalization and the stemming combination outperformed the baseline. Truncation had the best performance. The main conclusion of the paper is that truncation, normalization and stemming enhanced retrieval effectiveness in comparison to the baseline. Further, normalization and stemming were not far below truncation. 相似文献
19.
Xiangji Huang Fuchun Peng Dale Schuurmans Nick Cercone Stephen E. Robertson 《Information Retrieval》2003,6(3-4):333-362
We propose a self-supervised word segmentation technique for text segmentation in Chinese information retrieval. This method combines the advantages of traditional dictionary based, character based and mutual information based approaches, while overcoming many of their shortcomings. Experiments on TREC data show this method is promising. Our method is completely language independent and unsupervised, which provides a promising avenue for constructing accurate multi-lingual or cross-lingual information retrieval systems that are flexible and adaptive. We find that although the segmentation accuracy of self-supervised segmentation is not as high as some other segmentation methods, it is enough to give good retrieval performance. It is commonly believed that word segmentation accuracy is monotonically related to retrieval performance in Chinese information retrieval. However, for Chinese, we find that the relationship between segmentation and retrieval performance is in fact nonmonotonic; that is, at around 70% word segmentation accuracy an over-segmentation phenomenon begins to occur which leads to a reduction in information retrieval performance. We demonstrate this effect by presenting an empirical investigation of information retrieval on Chinese TREC data, using a wide variety of word segmentation algorithms with word segmentation accuracies ranging from 44% to 95%, including 70% word segmentation accuracy from our self-supervised word-segmentation approach. It appears that the main reason for the drop in retrieval performance is that correct compounds and collocations are preserved by accurate segmenters, while they are broken up by less accurate (but reasonable) segmenters, to a surprising advantage. This suggests that words themselves might be too broad a notion to conveniently capture the general semantic meaning of Chinese text. Our research suggests machine learning techniques can play an important role in building adaptable information retrieval systems and different evaluation standards for word segmentation should be given to different applications. 相似文献
20.
《中国分类主题词表》的结构及功能评介张强Ａｂｓｔｒａｃｔ：Ｔｈｅ＂ＣｈｉｎｅｓｅＣｌａｓｓｉｆｉｅｄｓｕｂｊｅｃｔＴｈｅｓａｕｒｕｓ＂ｗｉｔｈｉｔｓｉｎｔｅｇｒａｔｉｏｎｏｆｃｌａｓｓｉｆｉｃａｔｉｏｎａｎｄｓｕｂｊｅｃｔｉｎｄｅｘｉｎｇｉｓＣｈｉｎ... 相似文献