Kelly, D.; Harper, D.J.; Landau, B.: Questionnaire mode effects in interactive information retrieval experiments (2008)
0.01
0.013613109 = product of:
0.027226217 = sum of:
0.027226217 = product of:
0.054452434 = sum of:
0.054452434 = weight(_text_:systems in 2029) [ClassicSimilarity], result of:
0.054452434 = score(doc=2029,freq=8.0), product of:
0.16037072 = queryWeight, product of:
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.052184064 = queryNorm
0.339541 = fieldWeight in 2029, product of:
2.828427 = tf(freq=8.0), with freq of:
8.0 = termFreq=8.0
3.0731742 = idf(docFreq=5561, maxDocs=44218)
0.0390625 = fieldNorm(doc=2029)
0.5 = coord(1/2)
0.5 = coord(1/2)
- Abstract
- The questionnaire is an important technique for gathering data from subjects during interactive information retrieval (IR) experiments. Research in survey methodology, public opinion polling and psychology has demonstrated a number of response biases and behaviors that subjects exhibit when responding to questionnaires. Furthermore, research in human-computer interaction has demonstrated that subjects tend to inflate their ratings of systems when completing usability questionnaires. In this study we investigate the relationship between questionnaire mode and subjects' responses to a usability questionnaire comprised of closed and open questions administered during an interactive IR experiment. Three questionnaire modes (pen-and-paper, electronic and interview) were explored with 51 subjects who used one of two information retrieval systems. Results showed that subjects' quantitative evaluations of systems were significantly lower in the interview mode than in the electronic mode. With respect to open questions, subjects in the interview mode used significantly more words than subjects in the pen-and-paper or electronic modes to communicate their responses, and communicated a significantly higher number of response units, even though the total number of unique response units was roughly the same across condition. Finally, results showed that subjects in the pen-and-paper mode were the most efficient in communicating their responses to open questions. These results suggest that researchers should use the interview mode to elicit responses to closed questions from subjects and either pen-and-paper or electronic modes to elicit responses to open questions.
- Footnote
- Beitrag eines Themenbereichs: Evaluation of Interactive Information Retrieval Systems
