Tang, R.; Solomon, P.: Use of relevance criteria across stages of document evaluation : on the complementarity of experimental and naturalistic studies (2001)
0.02
0.016239362 = product of:
0.032478724 = sum of:
0.032478724 = product of:
0.06495745 = sum of:
0.06495745 = weight(_text_:90 in 5213) [ClassicSimilarity], result of:
0.06495745 = score(doc=5213,freq=2.0), product of:
0.2733978 = queryWeight, product of:
5.376119 = idf(docFreq=555, maxDocs=44218)
0.050854117 = queryNorm
0.23759314 = fieldWeight in 5213, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
5.376119 = idf(docFreq=555, maxDocs=44218)
0.03125 = fieldNorm(doc=5213)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- Tang and Solomon, based upon their review of the history of topical and other than topical criteria in relevance evaluation, decide to look at a two stage model where judgements are first made on surrogate records and then on full document text to determine if a criteria shift takes place and if so in what manner and to what degree. Both a controlled experiment and a naturalistic study were used to study the staging of relevance judgement criteria. In the controlled environment 90 undergraduate Psychology students were instructed to choose papers that would help them meet an assignment from 20 preselected papers on broader topic that included that assigned. They first selected on the basis of citation and abstract, then read the papers, and in each process filled out a questionnaire on the importance of each of 15 criteria at each stage of the two-stage process. In the naturalistic study 9 Ph.D. Psychology students conducted literature searches to support their own research and were asked to think aloud while making their decisions from retrieved surrogates, and later filled out a questionnaire while reading those materials that they selected and then interviewed at the end of the process. Apparently understandability is important at both stages. Importance increased at stage two. Cognitive criteria do not all follow the same pattern across stages. The controlled group thought quality of information was most important in stage one and topicality most important in stage 2. In the naturalistic study topicality was most frequent for stage one and research structure for stage two. A classification of criteria by their functionality is suggested as a better approach. First a division as to whether a criterion is objectively associated with the document as opposed to being subjectively associated with a person's expectations; then a division based on primary (essential) or secondary (for assistance) status.
