Document (#37395)

Author

Bilal, D.

Title

Ranking, relevance judgment, and precision of information retrieval on children's queries : evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

Source

Journal of the American Society for Information Science and Technology. 63(2012) no.9, S.1879-1896

Year

2012

Abstract

This study employed benchmarking and intellectual relevance judgment in evaluating Google, Yahoo!, Bing, Yahoo! Kids, and Ask Kids on 30 queries that children formulated to find information for specific tasks. Retrieved hits on given queries were benchmarked to Google's and Yahoo! Kids' top-five ranked hits retrieved. Relevancy of hits was judged on a graded scale; precision was calculated using the precision-at-ten metric (P@10). Yahoo! and Bing produced a similar percentage in hit overlap with Google (nearly 30%), but differed in the ranking of hits. Ask Kids retrieved 11% in hit overlap with Google versus 3% by Yahoo! Kids. The engines retrieved 26 hits across query clusters that overlapped with Yahoo! Kids' top-five ranked hits. Precision (P) that the engines produced across the queries was P = 0.48 for relevant hits, and P = 0.28 for partially relevant hits. Precision by Ask Kids was P = 0.44 for relevant hits versus P = 0.21 by Yahoo! Kids. Bing produced the highest total precision (TP) of relevant hits (TP = 0.86) across the queries, and Yahoo! Kids yielded the lowest (TP = 0.47). Average precision (AP) of relevant hits was AP = 0.56 by leading engines versus AP = 0.29 by small engines. In contrast, average precision of partially relevant hits was AP = 0.83 by small engines versus AP = 0.33 by leading engines. Average precision of relevant hits across the engines was highest on two-word queries and lowest on one-word queries. Google performed best on natural language queries; Bing did the same (P = 0.69) on two-word queries. The findings have implications for search engine ranking algorithms, relevance theory, search engine design, research design, and information literacy.

Theme

Retrievalalgorithmen
Suchmaschinen

Object

Google
Yahoo!
Bing
Yahoo! Kids
ask Kids

Similar documents (author)

1. Bilal, D.: Children's use of the Yahooligans! Web search engine : I. Cognitive, physical, and affective behaviors on fact-based search tasks (2000) 5.64

   5.639896 = sum of:
     5.639896 = weight(author_txt:bilal in 5777) [ClassicSimilarity], result of:
       5.639896 = fieldWeight in 5777, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.023833 = idf(docFreq=13, maxDocs=42740)
         0.625 = fieldNorm(doc=5777)
   

2. Bilal, D.: Children's use of the Yahooligans! Web search engine : II. Cognitive and physical behaviors on research tasks (2001) 5.64

   5.639896 = sum of:
     5.639896 = weight(author_txt:bilal in 499) [ClassicSimilarity], result of:
       5.639896 = fieldWeight in 499, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.023833 = idf(docFreq=13, maxDocs=42740)
         0.625 = fieldNorm(doc=499)
   

3. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 5.64

   5.639896 = sum of:
     5.639896 = weight(author_txt:bilal in 545) [ClassicSimilarity], result of:
       5.639896 = fieldWeight in 545, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.023833 = idf(docFreq=13, maxDocs=42740)
         0.625 = fieldNorm(doc=545)
   

4. Bilal, D.: Children's use of the Yahooligans! Web search engine : III. Cognitive and physical behaviors on fully self-generated search tasks (2002) 5.64

   5.639896 = sum of:
     5.639896 = weight(author_txt:bilal in 229) [ClassicSimilarity], result of:
       5.639896 = fieldWeight in 229, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.023833 = idf(docFreq=13, maxDocs=42740)
         0.625 = fieldNorm(doc=229)
   

5. Bilal, D.: Percentile ranks and the integrated impact indicator (I3) (2012) 5.64

   5.639896 = sum of:
     5.639896 = weight(author_txt:bilal in 2393) [ClassicSimilarity], result of:
       5.639896 = fieldWeight in 2393, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.023833 = idf(docFreq=13, maxDocs=42740)
         0.625 = fieldNorm(doc=2393)
   

Similar documents (content)

1. Bilal, D.: Web search engines for children : a comparative study and performance evaluation of Yahooligans!, AskJeeves for Kids, and Super Snooper (1999) 0.41

   0.41367492 = sum of:
     0.41367492 = product of:
       1.2927341 = sum of:
         0.027977291 = weight(abstract_txt:highest in 545) [ClassicSimilarity], result of:
           0.027977291 = score(doc=545,freq=1.0), product of:
             0.06502786 = queryWeight, product of:
               1.2061859 = boost
               6.883767 = idf(docFreq=118, maxDocs=42740)
               0.007831754 = queryNorm
             0.43023545 = fieldWeight in 545, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.883767 = idf(docFreq=118, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.028723 = weight(abstract_txt:overlap in 545) [ClassicSimilarity], result of:
           0.028723 = score(doc=545,freq=1.0), product of:
             0.06617829 = queryWeight, product of:
               1.2168087 = boost
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.007831754 = queryNorm
             0.43402448 = fieldWeight in 545, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.021857474 = weight(abstract_txt:relevance in 545) [ClassicSimilarity], result of:
           0.021857474 = score(doc=545,freq=2.0), product of:
             0.050116736 = queryWeight, product of:
               1.2968853 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.007831754 = queryNorm
             0.43613124 = fieldWeight in 545, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.031842127 = weight(abstract_txt:retrieved in 545) [ClassicSimilarity], result of:
           0.031842127 = score(doc=545,freq=1.0), product of:
             0.089311436 = queryWeight, product of:
               1.9990933 = boost
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.007831754 = queryNorm
             0.35652912 = fieldWeight in 545, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.07740154 = weight(abstract_txt:engines in 545) [ClassicSimilarity], result of:
           0.07740154 = score(doc=545,freq=3.0), product of:
             0.13491006 = queryWeight, product of:
               3.2502797 = boost
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.007831754 = queryNorm
             0.57372695 = fieldWeight in 545, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.050873414 = weight(abstract_txt:queries in 545) [ClassicSimilarity], result of:
           0.050873414 = score(doc=545,freq=1.0), product of:
             0.15994124 = queryWeight, product of:
               4.012832 = boost
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.007831754 = queryNorm
             0.31807566 = fieldWeight in 545, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.23857285 = weight(abstract_txt:hits in 545) [ClassicSimilarity], result of:
           0.23857285 = score(doc=545,freq=1.0), product of:
             0.5065405 = queryWeight, product of:
               8.582786 = boost
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.007831754 = queryNorm
             0.47098476 = fieldWeight in 545, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
         0.81548643 = weight(abstract_txt:kids in 545) [ClassicSimilarity], result of:
           0.81548643 = score(doc=545,freq=5.0), product of:
             0.6357764 = queryWeight, product of:
               8.845005 = boost
               9.177984 = idf(docFreq=11, maxDocs=42740)
               0.007831754 = queryNorm
             1.2826623 = fieldWeight in 545, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               9.177984 = idf(docFreq=11, maxDocs=42740)
               0.0625 = fieldNorm(doc=545)
       0.32 = coord(8/25)
   

2. Eastman, C.M.: 30,000 hits may be better than 300 : precision anomalies in Internet searches (2002) 0.22

   0.22052221 = sum of:
     0.22052221 = product of:
       0.7875793 = sum of:
         0.031722717 = weight(abstract_txt:ranked in 232) [ClassicSimilarity], result of:
           0.031722717 = score(doc=232,freq=2.0), product of:
             0.056121934 = queryWeight, product of:
               1.120549 = boost
               6.3950324 = idf(docFreq=193, maxDocs=42740)
               0.007831754 = queryNorm
             0.56524634 = fieldWeight in 232, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.3950324 = idf(docFreq=193, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.015455568 = weight(abstract_txt:relevance in 232) [ClassicSimilarity], result of:
           0.015455568 = score(doc=232,freq=1.0), product of:
             0.050116736 = queryWeight, product of:
               1.2968853 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.007831754 = queryNorm
             0.30839136 = fieldWeight in 232, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.039114244 = weight(abstract_txt:ranking in 232) [ClassicSimilarity], result of:
           0.039114244 = score(doc=232,freq=3.0), product of:
             0.06453208 = queryWeight, product of:
               1.4716277 = boost
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.007831754 = queryNorm
             0.60612094 = fieldWeight in 232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.5991054 = idf(docFreq=429, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.044687796 = weight(abstract_txt:engines in 232) [ClassicSimilarity], result of:
           0.044687796 = score(doc=232,freq=1.0), product of:
             0.13491006 = queryWeight, product of:
               3.2502797 = boost
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.007831754 = queryNorm
             0.3312414 = fieldWeight in 232, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.07194587 = weight(abstract_txt:queries in 232) [ClassicSimilarity], result of:
           0.07194587 = score(doc=232,freq=2.0), product of:
             0.15994124 = queryWeight, product of:
               4.012832 = boost
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.007831754 = queryNorm
             0.4498269 = fieldWeight in 232, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.17143285 = weight(abstract_txt:precision in 232) [ClassicSimilarity], result of:
           0.17143285 = score(doc=232,freq=7.0), product of:
             0.1879302 = queryWeight, product of:
               4.3497977 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.007831754 = queryNorm
             0.9122156 = fieldWeight in 232, product of:
               2.6457512 = tf(freq=7.0), with freq of:
                 7.0 = termFreq=7.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
         0.41322026 = weight(abstract_txt:hits in 232) [ClassicSimilarity], result of:
           0.41322026 = score(doc=232,freq=3.0), product of:
             0.5065405 = queryWeight, product of:
               8.582786 = boost
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.007831754 = queryNorm
             0.8157695 = fieldWeight in 232, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.0625 = fieldNorm(doc=232)
       0.28 = coord(7/25)
   

3. Li, L.; Shang, Y.; Zhang, W.: Improvement of HITS-based algorithms on Web documents 0.18

   0.17937449 = sum of:
     0.17937449 = product of:
       1.1210905 = sum of:
         0.03346229 = weight(abstract_txt:relevance in 4515) [ClassicSimilarity], result of:
           0.03346229 = score(doc=4515,freq=3.0), product of:
             0.050116736 = queryWeight, product of:
               1.2968853 = boost
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.007831754 = queryNorm
             0.66768694 = fieldWeight in 4515, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.934262 = idf(docFreq=835, maxDocs=42740)
               0.078125 = fieldNorm(doc=4515)
         0.06359177 = weight(abstract_txt:queries in 4515) [ClassicSimilarity], result of:
           0.06359177 = score(doc=4515,freq=1.0), product of:
             0.15994124 = queryWeight, product of:
               4.012832 = boost
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.007831754 = queryNorm
             0.39759457 = fieldWeight in 4515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.078125 = fieldNorm(doc=4515)
         0.08099441 = weight(abstract_txt:precision in 4515) [ClassicSimilarity], result of:
           0.08099441 = score(doc=4515,freq=1.0), product of:
             0.1879302 = queryWeight, product of:
               4.3497977 = boost
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.007831754 = queryNorm
             0.43098137 = fieldWeight in 4515, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5165615 = idf(docFreq=466, maxDocs=42740)
               0.078125 = fieldNorm(doc=4515)
         0.94304204 = weight(abstract_txt:hits in 4515) [ClassicSimilarity], result of:
           0.94304204 = score(doc=4515,freq=10.0), product of:
             0.5065405 = queryWeight, product of:
               8.582786 = boost
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.007831754 = queryNorm
             1.8617308 = fieldWeight in 4515, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.078125 = fieldNorm(doc=4515)
       0.16 = coord(4/25)
   

4. Bar-Ilan, J.: Evaluating the stability of the search tools Hotbot and Snap : a case study (2000) 0.15

   0.14805578 = sum of:
     0.14805578 = product of:
       0.7402789 = sum of:
         0.0430845 = weight(abstract_txt:overlap in 2181) [ClassicSimilarity], result of:
           0.0430845 = score(doc=2181,freq=1.0), product of:
             0.06617829 = queryWeight, product of:
               1.2168087 = boost
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.007831754 = queryNorm
             0.65103674 = fieldWeight in 2181, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.09375 = fieldNorm(doc=2181)
         0.04776319 = weight(abstract_txt:retrieved in 2181) [ClassicSimilarity], result of:
           0.04776319 = score(doc=2181,freq=1.0), product of:
             0.089311436 = queryWeight, product of:
               1.9990933 = boost
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.007831754 = queryNorm
             0.5347937 = fieldWeight in 2181, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.09375 = fieldNorm(doc=2181)
         0.0670317 = weight(abstract_txt:engines in 2181) [ClassicSimilarity], result of:
           0.0670317 = score(doc=2181,freq=1.0), product of:
             0.13491006 = queryWeight, product of:
               3.2502797 = boost
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.007831754 = queryNorm
             0.4968621 = fieldWeight in 2181, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.09375 = fieldNorm(doc=2181)
         0.07631012 = weight(abstract_txt:queries in 2181) [ClassicSimilarity], result of:
           0.07631012 = score(doc=2181,freq=1.0), product of:
             0.15994124 = queryWeight, product of:
               4.012832 = boost
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.007831754 = queryNorm
             0.4771135 = fieldWeight in 2181, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.09375 = fieldNorm(doc=2181)
         0.5060894 = weight(abstract_txt:hits in 2181) [ClassicSimilarity], result of:
           0.5060894 = score(doc=2181,freq=2.0), product of:
             0.5065405 = queryWeight, product of:
               8.582786 = boost
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.007831754 = queryNorm
             0.9991095 = fieldWeight in 2181, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.535756 = idf(docFreq=61, maxDocs=42740)
               0.09375 = fieldNorm(doc=2181)
       0.2 = coord(5/25)
   

5. Spink, A.; Jansen, B.J.; Blakely, C.; Koshman, S.: ¬A study of results overlap and uniqueness among major Web search engines (2006) 0.15

   0.14795 = sum of:
     0.14795 = product of:
       0.52839285 = sum of:
         0.050265245 = weight(abstract_txt:overlap in 2994) [ClassicSimilarity], result of:
           0.050265245 = score(doc=2994,freq=4.0), product of:
             0.06617829 = queryWeight, product of:
               1.2168087 = boost
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.007831754 = queryNorm
             0.7595428 = fieldWeight in 2994, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.9443917 = idf(docFreq=111, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.029241012 = weight(abstract_txt:across in 2994) [ClassicSimilarity], result of:
           0.029241012 = score(doc=2994,freq=2.0), product of:
             0.07320689 = queryWeight, product of:
               1.8099037 = boost
               5.1646085 = idf(docFreq=663, maxDocs=42740)
               0.007831754 = queryNorm
             0.3994298 = fieldWeight in 2994, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1646085 = idf(docFreq=663, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.039402623 = weight(abstract_txt:retrieved in 2994) [ClassicSimilarity], result of:
           0.039402623 = score(doc=2994,freq=2.0), product of:
             0.089311436 = queryWeight, product of:
               1.9990933 = boost
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.007831754 = queryNorm
             0.4411823 = fieldWeight in 2994, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.704466 = idf(docFreq=386, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.029379526 = weight(abstract_txt:google in 2994) [ClassicSimilarity], result of:
           0.029379526 = score(doc=2994,freq=1.0), product of:
             0.09967055 = queryWeight, product of:
               2.3611195 = boost
               5.3900137 = idf(docFreq=529, maxDocs=42740)
               0.007831754 = queryNorm
             0.29476637 = fieldWeight in 2994, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.3900137 = idf(docFreq=529, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.14630564 = weight(abstract_txt:engines in 2994) [ClassicSimilarity], result of:
           0.14630564 = score(doc=2994,freq=14.0), product of:
             0.13491006 = queryWeight, product of:
               3.2502797 = boost
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.007831754 = queryNorm
             1.0844679 = fieldWeight in 2994, product of:
               3.7416575 = tf(freq=14.0), with freq of:
                 14.0 = termFreq=14.0
               5.2998624 = idf(docFreq=579, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.12590528 = weight(abstract_txt:queries in 2994) [ClassicSimilarity], result of:
           0.12590528 = score(doc=2994,freq=8.0), product of:
             0.15994124 = queryWeight, product of:
               4.012832 = boost
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.007831754 = queryNorm
             0.78719705 = fieldWeight in 2994, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               5.0892105 = idf(docFreq=715, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
         0.10789352 = weight(abstract_txt:yahoo in 2994) [ClassicSimilarity], result of:
           0.10789352 = score(doc=2994,freq=1.0), product of:
             0.29891238 = queryWeight, product of:
               5.7825775 = boost
               6.6002955 = idf(docFreq=157, maxDocs=42740)
               0.007831754 = queryNorm
             0.36095366 = fieldWeight in 2994, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.6002955 = idf(docFreq=157, maxDocs=42740)
               0.0546875 = fieldNorm(doc=2994)
       0.28 = coord(7/25)