共查询到20条相似文献，搜索用时 750 毫秒
1.
Arthur R. Taylor Colleen Cool Nicholas J. Belkin William J. Amadio 《Information processing & management》2007
The process of information seeking involves a varied set of tasks and interactions. Exactly how the information seeker judges the relevance of what is retrieved has been a renewed area of interest in information retrieval studies. Various studies have identified facets or categories of relevance which go beyond simple topical relevance, and there has been some recent research on how these multi-dimensional concepts of relevance relate to the information seeking process. This study extends research on the relationship between multi-dimensional user relevance assignments and stage in the process of completing a task. Our results concur with and add detail to previous studies and suggest that users consistently identify relevance criteria beyond topical relevance. Our results also find a statistically significant relationship between the users’ stage in the search process and relevance categories chosen using the convenience sample chosen for this study. 相似文献
2.
Niall Rooney David Patterson Mykola Galushka Vladimir Dobrynin 《Information processing & management》2006
Contextual document clustering is a novel approach which uses information theoretic measures to cluster semantically related documents bound together by an implicit set of concepts or themes of narrow specificity. It facilitates cluster-based retrieval by assessing the similarity between a query and the cluster themes’ probability distribution. In this paper, we assess a relevance feedback mechanism, based on query refinement, that modifies the query’s probability distribution using a small number of documents that have been judged relevant to the query. We demonstrate that by providing only one relevance judgment, a performance improvement of 33% was obtained. 相似文献
3.
Ruihua Song Qingwei Guo Ruochi Zhang Guomao Xin Ji-Rong Wen Yong Yu Hsiao-Wuen Hon 《Information processing & management》2011
In the traditional evaluation of information retrieval systems, assessors are asked to determine the relevance of a document on a graded scale, independent of any other documents. Such judgments are absolute judgments. Learning to rank brings some new challenges to this traditional evaluation methodology, especially regarding absolute relevance judgments. Recently preferences judgments have been investigated as an alternative. Instead of assigning a relevance grade to a document, an assessor looks at a pair of pages and judges which one is better. In this paper, we generalize pairwise preference judgments to relative judgments. We formulate the problem of relative judgments in a formal way and then propose a new strategy called Select-the-Best-Ones to solve the problem. Through user studies, we compare our proposed method with a pairwise preference judgment method and an absolute judgment method. The results indicate that users can distinguish by about one more relevance degree when using relative methods than when using the absolute method. Consequently, the relative methods generate 15–30% more document pairs for learning to rank. Compared to the pairwise method, our proposed method increases the agreement among assessors from 95% to 99%, while halving the labeling time and the number of discordant pairs to experts’ judgments. 相似文献
4.
《Information processing & management》2023,60(2):103177
In order to evaluate the effectiveness of Information Retrieval (IR) systems it is key to collect relevance judgments from human assessors. Crowdsourcing has successfully been used as a method to scale-up the collection of manual relevance judgments, and previous research has investigated the impact of different judgment task design elements (e.g., highlighting query keywords in the document) on judgment quality and efficiency. In this work we investigate the positive and negative impacts of presenting crowd human assessors with more than just the topic and the document to be judged. We deploy different variants of crowdsourced relevance judgment tasks following a between-subjects design in which we present different types of metadata to the human assessor. Specifically, we investigate the effect of human metadata (e.g., what other human assessors think of the current document, as in which relevance level has already been selected by the majority crowd workers), machine metadata (e.g., how IR systems scored this document such as its average position in ranked lists, statistics about the document such as term frequencies). We look at the impact of metadata on judgment quality (i.e., the level of agreement with trained assessors) and cost (i.e., the time it takes for workers to complete the judgments) as well as at how metadata quality positively or negatively impact the collected judgments. 相似文献
5.
In this paper, we present the state of the art in the field of information retrieval that is relevant for understanding how to design information retrieval systems for children. We describe basic theories of human development to explain the specifics of young users, i.e., their cognitive skills, fine motor skills, knowledge, memory and emotional states in so far as they differ from those of adults. We derive the implications these differences have on the design of information retrieval systems for children. Furthermore, we summarize the main findings about children’s search behavior from multiple user studies. These findings are important to understand children’s information needs, their search strategies and usage of information retrieval systems. We also identify several weaknesses of previous user studies about children’s information-seeking behavior. Guided by the findings of these user studies, we describe challenges for the design of information retrieval systems for young users. We give an overview of algorithms and user interface concepts. We also describe existing information retrieval systems for children, in specific web search engines and digital libraries. We conclude with a discussion of open issues and directions for further research. The survey provided in this paper is important both for designers of information retrieval systems for young users as well as for researchers who start working in this field. 相似文献
6.
The relevance feedback process uses information obtained from a user about a set of initially retrieved documents to improve subsequent search formulations and retrieval performance. In extended Boolean models, the relevance feedback implies not only that new query terms must be identified and re-weighted, but also that the terms must be connected with Boolean And/Or operators properly. Salton et al. proposed a relevance feedback method, called DNF (disjunctive normal form) method, for a well established extended Boolean model. However, this method mainly focuses on generating Boolean queries but does not concern about re-weighting query terms. Also, this method has some problems in generating reformulated Boolean queries. In this study, we investigate the problems of the DNF method and propose a relevance feedback method using hierarchical clustering techniques to solve those problems. We also propose a neural network model in which the term weights used in extended Boolean queries can be adjusted by the users’ relevance feedbacks. 相似文献
7.
A new approach to the solicitation and measurement of relevance judgments is presented, which attempts to resolve some of the difficulties inherent in the nature of relevance and human judgment, and which further seeks to examine how users' judgments of document representations change as more information about documents is revealed to them. Subjects (university faculty and doctoral students) viewed three incremental versions of documents, and recorded ratio-level relevance judgments for each version. These judgments were analyzed by a variety of methods, including graphical inspection and examination of the number and degree of changes of judgments as new information is seen. A post questionnaire was also administered to obtain subjects' perceptions of the process and the individual fields of information presented. A consistent pattern of perception and importance of these fields is seen: Abstracts are by far the most important field and have the greatest impact, followed by titles, bibliographic information, and indexing. 相似文献
8.
Rocchio relevance feedback and latent semantic indexing (LSI) are well-known extensions of the vector space model for information retrieval (IR). This paper analyzes the statistical relationship between these extensions. The analysis focuses on each method’s basis in least-squares optimization. Noting that LSI and Rocchio relevance feedback both alter the vector space model in a way that is in some sense least-squares optimal, we ask: what is the relationship between LSI’s and Rocchio’s notions of optimality? What does this relationship imply for IR? Using an analytical approach, we argue that Rocchio relevance feedback is optimal if we understand retrieval as a simplified classification problem. On the other hand, LSI’s motivation comes to the fore if we understand it as a biased regression technique, where projection onto a low-dimensional orthogonal subspace of the documents reduces model variance. 相似文献
9.
Concurrent concepts of specificity are discussed and differentiated from each other to investigate the relationship between index term specificity and users’ relevance judgments. The identified concepts are term-document specificity, hierarchical specificity, statement specificity, and posting specificity. Among them, term-document specificity, which is a relationship between an index term and the document indexed with the term, is regarded as a fruitful research area. In an experiment involving three searches with 175 retrieved documents from 356 matched index terms, the impact of specificity on relevance judgments is analyzed and found to be statistically significant. Implications for index practice and for future research are discussed. 相似文献
10.
Michael J. Cole Jacek Gwizdka Chang Liu Nicholas J. Belkin Xiangmin Zhang 《Information processing & management》2013
The acquisition of information and the search interaction process is influenced strongly by a person’s use of their knowledge of the domain and the task. In this paper we show that a user’s level of domain knowledge can be inferred from their interactive search behaviors without considering the content of queries or documents. A technique is presented to model a user’s information acquisition process during search using only measurements of eye movement patterns. In a user study (n = 40) of search in the domain of genomics, a representation of the participant’s domain knowledge was constructed using self-ratings of knowledge of genomics-related terms (n = 409). Cognitive effort features associated with reading eye movement patterns were calculated for each reading instance during the search tasks. The results show correlations between the cognitive effort due to reading and an individual’s level of domain knowledge. We construct exploratory regression models that suggest it is possible to build models that can make predictions of the user’s level of knowledge based on real-time measurements of eye movement patterns during a task session. 相似文献
11.
With over 60,000 US K-12 science and mathematics education standards and a rapid proliferation of Web-enabled curriculum, retrieving curriculum that aligns with the standards to which teachers must teach is a key objective for educational digital libraries. However, previous studies of such alignment use single-dimensional and binary measures of the alignment concept. As a consequence, they suffer from low inter-rater reliability (IRR), with experts agreeing about alignments only some 20–40% of the time. We present the results of an experiment in which the alignment variable was operationalized using the Saracevic model of relevance ‘clues’ taken from the everyday practice of K-12 teaching. Results show high IRR across all clues with IRR on several specific alignment dimensions significantly higher than on overall alignment. In addition, a model of overall alignment is derived and estimated. The structure and explanatory power of the model as well as the relationships between alignment clues differ significantly between alignments of curriculum found by users themselves and curriculum found by others. These results illustrate the usefulness of clue-based relevance measures for information retrieval and have important consequences for both the formulation of automated retrieval mechanisms and the construction of a gold standard or benchmark set of standard-curriculum alignments. 相似文献
12.
Linda Schamber Michael B. Eisenberg Michael S. Nilan 《Information processing & management》1990,26(6)
Although relevance judgments are fundamental to the design and evaluation of all information retrieval systems, information scientists have not reached a consensus in defining the central concept of relevance. In this paper we ask two questions: What is the meaning of relevance? and What role does relevance play in information behavior? We attempt to address these questions by reviewing literature over the last 30 years that presents various views of relevance as topical, user-oriented, multidimensional, cognitive, and dynamic. We then discuss traditional assumptions on which most research in the field has been based and begin building a case for an approach to the problem of definition based on alternative assumptions. The dynamic, situational approach we suggest views the user — regardless of system — as the central and active determinant of the dimensions of relevance. We believe that relevance is a multidimensional concept; that it is dependent on both internal (cognitive) and external (situational) factors; that it is based on a dynamic human judgment process; and that it is a complex but systematic and measurable phenomenon. 相似文献
13.
Search engine researchers typically depict search as the solitary activity of an individual searcher. In contrast, results from our critical-incident survey of 150 users on Amazon’s Mechanical Turk service suggest that social interactions play an important role throughout the search process. A second survey of also 150 users, focused instead on difficulties encountered during searches, suggests similar conclusions. These social interactions range from highly coordinated collaborations with shared goals to loosely coordinated collaborations in which only advice is sought. Our main contribution is that we have integrated models from previous work in sensemaking and information-seeking behavior to present a canonical social model of user activities before, during, and after a search episode, suggesting where in the search process both explicitly and implicitly shared information may be valuable to individual searchers. 相似文献
14.
15.
16.
The evaluation of exploratory search relies on the ongoing paradigm shift from focusing on the search algorithm to focusing on the interactive process. This paper proposes a model-driven formative evaluation approach, in which the goal is not the evaluation of a specific system, per se, but the exploration of new design possibilities. This paper gives an example of this approach where a model of sensemaking was used to inform the evaluation of a basic exploratory search system(s) in the context of a sensemaking task. The model suggested that, rather than just looking at simple search performance measures, we should examine closely the interwoven, interactive processes of both representation construction and information seeking. Participants were asked to make sense of an unfamiliar topic using an augmented query-based search system. The processes of representation construction and information seeking were captured and analyzed using data from experiment notes, interviews, and a system log. The data analysis revealed users’ sources of ideas for structuring representations and a tightly coupled relationship between search and representation construction in their exploratory searches. For example, users strategically used search to find useful structure ideas instead of just accumulating information facts. Implications for improving current search systems and designing new systems are discussed. 相似文献
17.
A new model for aggregating multiple criteria evaluations for relevance assessment is proposed. An Information Retrieval context is considered, where relevance is modeled as a multidimensional property of documents. The usefulness and effectiveness of such a model are demonstrated by means of a case study on personalized Information Retrieval with multi-criteria relevance. The following criteria are considered to estimate document relevance: aboutness, coverage, appropriateness, and reliability. 相似文献
18.
Current awareness services are designed to keep users informed about recent developments based around user need profiles. In organisational settings, they may operate through both electronic and social interactions aimed at delivering information that is relevant, pertinent and current. Understanding these interactions can reveal the tensions in current awareness dissemination and help inform ways of making services more effective and efficient. We report an in-depth, observational study of electronic current awareness use within a large London law firm. The study found that selection, re-aggregation and forwarding of information by multiple actors gives rise to a complex sociotechnical distribution network. Knowledge management staff act as a layer of “intelligent filters” sensitive to complex, local information needs; their distribution decisions address multiple situational relevance factors in a situation fraught with information overload and restrictive time-pressures. Their decisions aim to optimise conflicting constraints of recall, precision and information quantity. Critical to this is the use of dynamic profile updates which propagate back through the network through formal and informal social interactions. This supports changes to situational relevance judgements and so allows the network to ‘self-tune’. These findings lead to design requirements, including that systems should support rapid assessment of information items against an individual’s interests; that it should be possible to organise information for different subsequent uses; and that there should be back-propagation from information consumers to providers, to tune the understanding of their information needs. 相似文献
19.
Lori Lorigo Bing Pan Helene Hembrooke Thorsten Joachims Laura Granka Geri Gay 《Information processing & management》2006
To improve search engine effectiveness, we have observed an increased interest in gathering additional feedback about users’ information needs that goes beyond the queries they type in. Adaptive search engines use explicit and implicit feedback indicators to model users or search tasks. In order to create appropriate models, it is essential to understand how users interact with search engines, including the determining factors of their actions. Using eye tracking, we extend this understanding by analyzing the sequences and patterns with which users evaluate query result returned to them when using Google. We find that the query result abstracts are viewed in the order of their ranking in only about one fifth of the cases, and only an average of about three abstracts per result page are viewed at all. We also compare search behavior variability with respect to different classes of users and different classes of search tasks to reveal whether user models or task models may be greater predictors of behavior. We discover that gender and task significantly influence different kinds of search behaviors discussed here. The results are suggestive of improvements to query-based search interface designs with respect to both their use of space and workflow. 相似文献