Document (#34044)

Author

Qu, Y.
Furnas, G.W.

Title

Model-driven formative evaluation of exploratory search : a study under a sensemaking framework

Source

Information processing and management. 44(2008) no.2, S.511-533

Year

2008

Abstract

The evaluation of exploratory search relies on the ongoing paradigm shift from focusing on the search algorithm to focusing on the interactive process. This paper proposes a model-driven formative evaluation approach, in which the goal is not the evaluation of a specific system, per se, but the exploration of new design possibilities. This paper gives an example of this approach where a model of sensemaking was used to inform the evaluation of a basic exploratory search system(s) in the context of a sensemaking task. The model suggested that, rather than just looking at simple search performance measures, we should examine closely the interwoven, interactive processes of both representation construction and information seeking. Participants were asked to make sense of an unfamiliar topic using an augmented query-based search system. The processes of representation construction and information seeking were captured and analyzed using data from experiment notes, interviews, and a system log. The data analysis revealed users' sources of ideas for structuring representations and a tightly coupled relationship between search and representation construction in their exploratory searches. For example, users strategically used search to find useful structure ideas instead of just accumulating information facts. Implications for improving current search systems and designing new systems are discussed.

Footnote

Beitrag eines Themenschwerpunktes "Evaluating exploratory search systems"

Similar documents (content)

1. Pontis, S.; Blandford, A.: Understanding "influence" : an exploratory study of academics' processes of knowledge construction through iterative and interactive information seeking (2015) 0.32

   0.32474762 = sum of:
     0.32474762 = product of:
       1.0148363 = sum of:
         0.09376514 = weight(abstract_txt:unfamiliar in 2126) [ClassicSimilarity], result of:
           0.09376514 = score(doc=2126,freq=2.0), product of:
             0.13406697 = queryWeight, product of:
               1.0563425 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016039561 = queryNorm
             0.6993903 = fieldWeight in 2126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.04992427 = weight(abstract_txt:seeking in 2126) [ClassicSimilarity], result of:
           0.04992427 = score(doc=2126,freq=2.0), product of:
             0.11096313 = queryWeight, product of:
               1.3590899 = boost
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.016039561 = queryNorm
             0.44991764 = fieldWeight in 2126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.051236298 = weight(abstract_txt:processes in 2126) [ClassicSimilarity], result of:
           0.051236298 = score(doc=2126,freq=2.0), product of:
             0.11289881 = queryWeight, product of:
               1.3708929 = boost
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.016039561 = queryNorm
             0.45382494 = fieldWeight in 2126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.045877483 = weight(abstract_txt:interactive in 2126) [ClassicSimilarity], result of:
           0.045877483 = score(doc=2126,freq=1.0), product of:
             0.13214394 = queryWeight, product of:
               1.4831411 = boost
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.016039561 = queryNorm
             0.3471781 = fieldWeight in 2126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5548496 = idf(docFreq=464, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.06515952 = weight(abstract_txt:construction in 2126) [ClassicSimilarity], result of:
           0.06515952 = score(doc=2126,freq=1.0), product of:
             0.19113044 = queryWeight, product of:
               2.1845872 = boost
               5.4546638 = idf(docFreq=513, maxDocs=44218)
               0.016039561 = queryNorm
             0.34091648 = fieldWeight in 2126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4546638 = idf(docFreq=513, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.44321558 = weight(abstract_txt:sensemaking in 2126) [ClassicSimilarity], result of:
           0.44321558 = score(doc=2126,freq=3.0), product of:
             0.4757523 = queryWeight, product of:
               3.446632 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.016039561 = queryNorm
             0.93161 = fieldWeight in 2126, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.2066997 = weight(abstract_txt:exploratory in 2126) [ClassicSimilarity], result of:
           0.2066997 = score(doc=2126,freq=2.0), product of:
             0.36047414 = queryWeight, product of:
               3.4642632 = boost
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.016039561 = queryNorm
             0.57341063 = fieldWeight in 2126, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
         0.058958292 = weight(abstract_txt:search in 2126) [ClassicSimilarity], result of:
           0.058958292 = score(doc=2126,freq=1.0), product of:
             0.2578784 = queryWeight, product of:
               4.3951383 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.016039561 = queryNorm
             0.22862828 = fieldWeight in 2126, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0625 = fieldNorm(doc=2126)
       0.32 = coord(8/25)
   

2. Zhang, P.; Soergel, D.: Towards a comprehensive model of the cognitive process and mechanisms of individual sensemaking (2014) 0.25

   0.24809134 = sum of:
     0.24809134 = product of:
       1.0337139 = sum of:
         0.03530179 = weight(abstract_txt:seeking in 1344) [ClassicSimilarity], result of:
           0.03530179 = score(doc=1344,freq=1.0), product of:
             0.11096313 = queryWeight, product of:
               1.3590899 = boost
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.016039561 = queryNorm
             0.31813982 = fieldWeight in 1344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
         0.0627514 = weight(abstract_txt:processes in 1344) [ClassicSimilarity], result of:
           0.0627514 = score(doc=1344,freq=3.0), product of:
             0.11289881 = queryWeight, product of:
               1.3708929 = boost
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.016039561 = queryNorm
             0.5558198 = fieldWeight in 1344, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
         0.047203053 = weight(abstract_txt:ideas in 1344) [ClassicSimilarity], result of:
           0.047203053 = score(doc=1344,freq=1.0), product of:
             0.13467726 = queryWeight, product of:
               1.4972901 = boost
               5.6078424 = idf(docFreq=440, maxDocs=44218)
               0.016039561 = queryNorm
             0.35049015 = fieldWeight in 1344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6078424 = idf(docFreq=440, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
         0.020530263 = weight(abstract_txt:system in 1344) [ClassicSimilarity], result of:
           0.020530263 = score(doc=1344,freq=1.0), product of:
             0.097406335 = queryWeight, product of:
               1.8008081 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.016039561 = queryNorm
             0.21076928 = fieldWeight in 1344, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
         0.058730226 = weight(abstract_txt:model in 1344) [ClassicSimilarity], result of:
           0.058730226 = score(doc=1344,freq=3.0), product of:
             0.13610004 = queryWeight, product of:
               2.1286435 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.016039561 = queryNorm
             0.4315225 = fieldWeight in 1344, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
         0.80919725 = weight(abstract_txt:sensemaking in 1344) [ClassicSimilarity], result of:
           0.80919725 = score(doc=1344,freq=10.0), product of:
             0.4757523 = queryWeight, product of:
               3.446632 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.016039561 = queryNorm
             1.7008793 = fieldWeight in 1344, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.0625 = fieldNorm(doc=1344)
       0.24 = coord(6/25)
   

3. White, R.W.; Roth, R.A.: Exploratory search : beyond the query-response paradigm (2009) 0.24

   0.24274251 = sum of:
     0.24274251 = product of:
       0.8669375 = sum of:
         0.066301964 = weight(abstract_txt:unfamiliar in 0) [ClassicSimilarity], result of:
           0.066301964 = score(doc=0,freq=1.0), product of:
             0.13406697 = queryWeight, product of:
               1.0563425 = boost
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.016039561 = queryNorm
             0.4945436 = fieldWeight in 0, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.912698 = idf(docFreq=43, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.061144494 = weight(abstract_txt:seeking in 0) [ClassicSimilarity], result of:
           0.061144494 = score(doc=0,freq=3.0), product of:
             0.11096313 = queryWeight, product of:
               1.3590899 = boost
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.016039561 = queryNorm
             0.55103433 = fieldWeight in 0, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.036229536 = weight(abstract_txt:processes in 0) [ClassicSimilarity], result of:
           0.036229536 = score(doc=0,freq=1.0), product of:
             0.11289881 = queryWeight, product of:
               1.3708929 = boost
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.016039561 = queryNorm
             0.3209027 = fieldWeight in 0, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.033907913 = weight(abstract_txt:model in 0) [ClassicSimilarity], result of:
           0.033907913 = score(doc=0,freq=1.0), product of:
             0.13610004 = queryWeight, product of:
               2.1286435 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.016039561 = queryNorm
             0.24913962 = fieldWeight in 0, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.060411613 = weight(abstract_txt:evaluation in 0) [ClassicSimilarity], result of:
           0.060411613 = score(doc=0,freq=1.0), product of:
             0.21546389 = queryWeight, product of:
               2.9944432 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016039561 = queryNorm
             0.2803793 = fieldWeight in 0, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.4133994 = weight(abstract_txt:exploratory in 0) [ClassicSimilarity], result of:
           0.4133994 = score(doc=0,freq=8.0), product of:
             0.36047414 = queryWeight, product of:
               3.4642632 = boost
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.016039561 = queryNorm
             1.1468213 = fieldWeight in 0, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
         0.19554254 = weight(abstract_txt:search in 0) [ClassicSimilarity], result of:
           0.19554254 = score(doc=0,freq=11.0), product of:
             0.2578784 = queryWeight, product of:
               4.3951383 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.016039561 = queryNorm
             0.75827426 = fieldWeight in 0, product of:
               3.3166249 = tf(freq=11.0), with freq of:
                 11.0 = termFreq=11.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0625 = fieldNorm(doc=0)
       0.28 = coord(7/25)
   

4. Blocks, D.; Cunliffe, D.; Tudhope, D.: ¬A reference model for user-system interaction in thesaurus-based searching (2006) 0.17

   0.16760263 = sum of:
     0.16760263 = product of:
       0.59858084 = sum of:
         0.07303037 = weight(abstract_txt:augmented in 202) [ClassicSimilarity], result of:
           0.07303037 = score(doc=202,freq=1.0), product of:
             0.1232253 = queryWeight, product of:
               1.0127302 = boost
               7.5860133 = idf(docFreq=60, maxDocs=44218)
               0.016039561 = queryNorm
             0.59265727 = fieldWeight in 202, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.5860133 = idf(docFreq=60, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.06404537 = weight(abstract_txt:processes in 202) [ClassicSimilarity], result of:
           0.06404537 = score(doc=202,freq=2.0), product of:
             0.11289881 = queryWeight, product of:
               1.3708929 = boost
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.016039561 = queryNorm
             0.5672812 = fieldWeight in 202, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.05738383 = weight(abstract_txt:system in 202) [ClassicSimilarity], result of:
           0.05738383 = score(doc=202,freq=5.0), product of:
             0.097406335 = queryWeight, product of:
               1.8008081 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.016039561 = queryNorm
             0.58911806 = fieldWeight in 202, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.059991956 = weight(abstract_txt:representation in 202) [ClassicSimilarity], result of:
           0.059991956 = score(doc=202,freq=1.0), product of:
             0.15588343 = queryWeight, product of:
               1.9728972 = boost
               4.926098 = idf(docFreq=871, maxDocs=44218)
               0.016039561 = queryNorm
             0.3848514 = fieldWeight in 202, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.926098 = idf(docFreq=871, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.10382136 = weight(abstract_txt:model in 202) [ClassicSimilarity], result of:
           0.10382136 = score(doc=202,freq=6.0), product of:
             0.13610004 = queryWeight, product of:
               2.1286435 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.016039561 = queryNorm
             0.7628312 = fieldWeight in 202, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.07551452 = weight(abstract_txt:evaluation in 202) [ClassicSimilarity], result of:
           0.07551452 = score(doc=202,freq=1.0), product of:
             0.21546389 = queryWeight, product of:
               2.9944432 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016039561 = queryNorm
             0.35047412 = fieldWeight in 202, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
         0.16479343 = weight(abstract_txt:search in 202) [ClassicSimilarity], result of:
           0.16479343 = score(doc=202,freq=5.0), product of:
             0.2578784 = queryWeight, product of:
               4.3951383 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.016039561 = queryNorm
             0.63903546 = fieldWeight in 202, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.078125 = fieldNorm(doc=202)
       0.28 = coord(7/25)
   

5. Shah, C.; Hendahewa, C.; González-Ibáñez, R.: Rain or shine? : forecasting search process performance in exploratory search tasks (2016) 0.16

   0.16413583 = sum of:
     0.16413583 = product of:
       0.5861994 = sum of:
         0.014062165 = weight(abstract_txt:approach in 3012) [ClassicSimilarity], result of:
           0.014062165 = score(doc=3012,freq=1.0), product of:
             0.060073417 = queryWeight, product of:
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.016039561 = queryNorm
             0.234083 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.745328 = idf(docFreq=2839, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.03530179 = weight(abstract_txt:seeking in 3012) [ClassicSimilarity], result of:
           0.03530179 = score(doc=3012,freq=1.0), product of:
             0.11096313 = queryWeight, product of:
               1.3590899 = boost
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.016039561 = queryNorm
             0.31813982 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.090237 = idf(docFreq=739, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.0627514 = weight(abstract_txt:processes in 3012) [ClassicSimilarity], result of:
           0.0627514 = score(doc=3012,freq=3.0), product of:
             0.11289881 = queryWeight, product of:
               1.3708929 = boost
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.016039561 = queryNorm
             0.5558198 = fieldWeight in 3012, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.1344433 = idf(docFreq=707, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.020530263 = weight(abstract_txt:system in 3012) [ClassicSimilarity], result of:
           0.020530263 = score(doc=3012,freq=1.0), product of:
             0.097406335 = queryWeight, product of:
               1.8008081 = boost
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.016039561 = queryNorm
             0.21076928 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3723085 = idf(docFreq=4123, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.060411613 = weight(abstract_txt:evaluation in 3012) [ClassicSimilarity], result of:
           0.060411613 = score(doc=3012,freq=1.0), product of:
             0.21546389 = queryWeight, product of:
               2.9944432 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016039561 = queryNorm
             0.2803793 = fieldWeight in 3012, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.2066997 = weight(abstract_txt:exploratory in 3012) [ClassicSimilarity], result of:
           0.2066997 = score(doc=3012,freq=2.0), product of:
             0.36047414 = queryWeight, product of:
               3.4642632 = boost
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.016039561 = queryNorm
             0.57341063 = fieldWeight in 3012, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.487401 = idf(docFreq=182, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
         0.1864425 = weight(abstract_txt:search in 3012) [ClassicSimilarity], result of:
           0.1864425 = score(doc=3012,freq=10.0), product of:
             0.2578784 = queryWeight, product of:
               4.3951383 = boost
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.016039561 = queryNorm
             0.7229861 = fieldWeight in 3012, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               3.6580524 = idf(docFreq=3098, maxDocs=44218)
               0.0625 = fieldNorm(doc=3012)
       0.28 = coord(7/25)