Document (#38106)

Author
Verberne, S.
Heijden, M. van der
Hinne, M.
Sappelli, M.
Koldijk, S.
Hoenkamp, E.
Kraaij, W.
Title
Reliability and validity of query intent assessments
Source
Journal of the American Society for Information Science and Technology. 64(2013) no.11, S.2224-2237
Year
2013
Abstract
In most intent recognition studies, annotations of query intent are created post hoc by external assessors who are not the searchers themselves. It is important for the field to get a better understanding of the quality of this process as an approximation for determining the searcher's actual intent. Some studies have investigated the reliability of the query intent annotation process by measuring the interassessor agreement. However, these studies did not measure the validity of the judgments, that is, to what extent the annotations match the searcher's actual intent. In this study, we asked both the searchers themselves and external assessors to classify queries using the same intent classification scheme. We show that of the seven dimensions in our intent classification scheme, four can reliably be used for query annotation. Of these four, only the annotations on the topic and spatial sensitivity dimension are valid when compared with the searcher's annotations. The difference between the interassessor agreement and the assessor-searcher agreement was significant on all dimensions, showing that the agreement between external assessors is not a good estimator of the validity of the intent classifications. Therefore, we encourage the research community to consider using query intent classifications by the searchers themselves as test data.
Theme
Suchtaktik

Similar documents (author)

  1. Kraaij, W.; Pohlmann, R.: Evaluation of a Dutch stemming algorithm (1995) 4.94
    4.941542 = sum of:
      4.941542 = weight(author_txt:kraaij in 6867) [ClassicSimilarity], result of:
        4.941542 = score(doc=6867,freq=1.0), product of:
          0.99999994 = queryWeight, product of:
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.101182975 = queryNorm
          4.9415426 = fieldWeight in 6867, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.5 = fieldNorm(doc=6867)
    
  2. Hiemstra, D.; Kraaij, W.: ¬A language-modeling approach to TREC (2005) 4.94
    4.941542 = sum of:
      4.941542 = weight(author_txt:kraaij in 92) [ClassicSimilarity], result of:
        4.941542 = score(doc=92,freq=1.0), product of:
          0.99999994 = queryWeight, product of:
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.101182975 = queryNorm
          4.9415426 = fieldWeight in 92, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.5 = fieldNorm(doc=92)
    
  3. Sappelli, M.; Verberne, S.; Kraaij, W.: Evaluation of context-aware recommendation systems for information re-finding (2017) 3.71
    3.7061567 = sum of:
      3.7061567 = weight(author_txt:kraaij in 4993) [ClassicSimilarity], result of:
        3.7061567 = score(doc=4993,freq=1.0), product of:
          0.99999994 = queryWeight, product of:
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.101182975 = queryNorm
          3.706157 = fieldWeight in 4993, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.375 = fieldNorm(doc=4993)
    
  4. Meij, E.; Trieschnigg, D.; Rijke, M. de; Kraaij, W.: Conceptual language models for domain-specific retrieval (2010) 3.09
    3.088464 = sum of:
      3.088464 = weight(author_txt:kraaij in 703) [ClassicSimilarity], result of:
        3.088464 = score(doc=703,freq=1.0), product of:
          0.99999994 = queryWeight, product of:
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.101182975 = queryNorm
          3.0884643 = fieldWeight in 703, product of:
            1.0 = tf(freq=1.0), with freq of:
              1.0 = termFreq=1.0
            9.883085 = idf(docFreq=5, maxDocs=43254)
            0.3125 = fieldNorm(doc=703)
    

Similar documents (content)

  1. Osman, D.J.; Yearwood, J.; Vamplew, P.: Automated opinion detection : implications of the level of agreement between human raters (2010) 0.13
    0.13474765 = sum of:
      0.13474765 = product of:
        0.67373824 = sum of:
          0.04771021 = weight(abstract_txt:assessments in 697) [ClassicSimilarity], result of:
            0.04771021 = score(doc=697,freq=3.0), product of:
              0.061982736 = queryWeight, product of:
                7.110497 = idf(docFreq=95, maxDocs=43254)
                0.008717075 = queryNorm
              0.76973385 = fieldWeight in 697, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                7.110497 = idf(docFreq=95, maxDocs=43254)
                0.0625 = fieldNorm(doc=697)
          0.0044124485 = weight(abstract_txt:that in 697) [ClassicSimilarity], result of:
            0.0044124485 = score(doc=697,freq=2.0), product of:
              0.02092765 = queryWeight, product of:
                1.0064344 = boost
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.008717075 = queryNorm
              0.210843 = fieldWeight in 697, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.0625 = fieldNorm(doc=697)
          0.010285831 = weight(abstract_txt:process in 697) [ClassicSimilarity], result of:
            0.010285831 = score(doc=697,freq=1.0), product of:
              0.040495478 = queryWeight, product of:
                1.1430964 = boost
                4.063992 = idf(docFreq=2019, maxDocs=43254)
                0.008717075 = queryNorm
              0.2539995 = fieldWeight in 697, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.063992 = idf(docFreq=2019, maxDocs=43254)
                0.0625 = fieldNorm(doc=697)
          0.43263373 = weight(abstract_txt:assessors in 697) [ClassicSimilarity], result of:
            0.43263373 = score(doc=697,freq=8.0), product of:
              0.28032443 = queryWeight, product of:
                3.683458 = boost
                8.730406 = idf(docFreq=18, maxDocs=43254)
                0.008717075 = queryNorm
              1.5433322 = fieldWeight in 697, product of:
                2.828427 = tf(freq=8.0), with freq of:
                  8.0 = termFreq=8.0
                8.730406 = idf(docFreq=18, maxDocs=43254)
                0.0625 = fieldNorm(doc=697)
          0.17869607 = weight(abstract_txt:agreement in 697) [ClassicSimilarity], result of:
            0.17869607 = score(doc=697,freq=3.0), product of:
              0.23729752 = queryWeight, product of:
                3.9132826 = boost
                6.956346 = idf(docFreq=111, maxDocs=43254)
                0.008717075 = queryNorm
              0.7530465 = fieldWeight in 697, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                6.956346 = idf(docFreq=111, maxDocs=43254)
                0.0625 = fieldNorm(doc=697)
        0.2 = coord(5/25)
    
  2. Smith, C.L.: Domain-independent search expertise : gaining knowledge in query formulation through guided practice (2017) 0.13
    0.13330294 = sum of:
      0.13330294 = product of:
        0.6665147 = sum of:
          0.007800181 = weight(abstract_txt:that in 5108) [ClassicSimilarity], result of:
            0.007800181 = score(doc=5108,freq=4.0), product of:
              0.02092765 = queryWeight, product of:
                1.0064344 = boost
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.008717075 = queryNorm
              0.37272128 = fieldWeight in 5108, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.078125 = fieldNorm(doc=5108)
          0.040375475 = weight(abstract_txt:dimensions in 5108) [ClassicSimilarity], result of:
            0.040375475 = score(doc=5108,freq=1.0), product of:
              0.08683977 = queryWeight, product of:
                1.6739365 = boost
                5.95126 = idf(docFreq=305, maxDocs=43254)
                0.008717075 = queryNorm
              0.4649422 = fieldWeight in 5108, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.95126 = idf(docFreq=305, maxDocs=43254)
                0.078125 = fieldNorm(doc=5108)
          0.061792783 = weight(abstract_txt:searchers in 5108) [ClassicSimilarity], result of:
            0.061792783 = score(doc=5108,freq=1.0), product of:
              0.1320168 = queryWeight, product of:
                2.5277834 = boost
                5.9912653 = idf(docFreq=293, maxDocs=43254)
                0.008717075 = queryNorm
              0.4680676 = fieldWeight in 5108, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.9912653 = idf(docFreq=293, maxDocs=43254)
                0.078125 = fieldNorm(doc=5108)
          0.12480435 = weight(abstract_txt:query in 5108) [ClassicSimilarity], result of:
            0.12480435 = score(doc=5108,freq=6.0), product of:
              0.13763313 = queryWeight, product of:
                3.332047 = boost
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.008717075 = queryNorm
              0.90679 = fieldWeight in 5108, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.078125 = fieldNorm(doc=5108)
          0.43174192 = weight(abstract_txt:intent in 5108) [ClassicSimilarity], result of:
            0.43174192 = score(doc=5108,freq=1.0), product of:
              0.72074187 = queryWeight, product of:
                10.78337 = boost
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.008717075 = queryNorm
              0.59902436 = fieldWeight in 5108, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.078125 = fieldNorm(doc=5108)
        0.2 = coord(5/25)
    
  3. Selvaretnam, B.; Belkhatir, M.: ¬A linguistically driven framework for query expansion via grammatical constituent highlighting and role-based concept weighting (2016) 0.10
    0.09963862 = sum of:
      0.09963862 = product of:
        0.6227414 = sum of:
          0.0067551546 = weight(abstract_txt:that in 4341) [ClassicSimilarity], result of:
            0.0067551546 = score(doc=4341,freq=3.0), product of:
              0.02092765 = queryWeight, product of:
                1.0064344 = boost
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.008717075 = queryNorm
              0.3227861 = fieldWeight in 4341, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.078125 = fieldNorm(doc=4341)
          0.03139083 = weight(abstract_txt:scheme in 4341) [ClassicSimilarity], result of:
            0.03139083 = score(doc=4341,freq=1.0), product of:
              0.07342469 = queryWeight, product of:
                1.53922 = boost
                5.4723096 = idf(docFreq=493, maxDocs=43254)
                0.008717075 = queryNorm
              0.42752418 = fieldWeight in 4341, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.4723096 = idf(docFreq=493, maxDocs=43254)
                0.078125 = fieldNorm(doc=4341)
          0.15285349 = weight(abstract_txt:query in 4341) [ClassicSimilarity], result of:
            0.15285349 = score(doc=4341,freq=9.0), product of:
              0.13763313 = queryWeight, product of:
                3.332047 = boost
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.008717075 = queryNorm
              1.1105864 = fieldWeight in 4341, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.078125 = fieldNorm(doc=4341)
          0.43174192 = weight(abstract_txt:intent in 4341) [ClassicSimilarity], result of:
            0.43174192 = score(doc=4341,freq=1.0), product of:
              0.72074187 = queryWeight, product of:
                10.78337 = boost
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.008717075 = queryNorm
              0.59902436 = fieldWeight in 4341, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.078125 = fieldNorm(doc=4341)
        0.16 = coord(4/25)
    
  4. White, R.W.; Jose, J.M.; Ruthven, I.: Using top-ranking sentences to facilitate effective information access (2005) 0.10
    0.09923201 = sum of:
      0.09923201 = product of:
        0.35440004 = sum of:
          0.0054041236 = weight(abstract_txt:that in 5882) [ClassicSimilarity], result of:
            0.0054041236 = score(doc=5882,freq=3.0), product of:
              0.02092765 = queryWeight, product of:
                1.0064344 = boost
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.008717075 = queryNorm
              0.25822887 = fieldWeight in 5882, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.010285831 = weight(abstract_txt:process in 5882) [ClassicSimilarity], result of:
            0.010285831 = score(doc=5882,freq=1.0), product of:
              0.040495478 = queryWeight, product of:
                1.1430964 = boost
                4.063992 = idf(docFreq=2019, maxDocs=43254)
                0.008717075 = queryNorm
              0.2539995 = fieldWeight in 5882, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.063992 = idf(docFreq=2019, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.035026114 = weight(abstract_txt:actual in 5882) [ClassicSimilarity], result of:
            0.035026114 = score(doc=5882,freq=1.0), product of:
              0.09165896 = queryWeight, product of:
                1.719757 = boost
                6.1141634 = idf(docFreq=259, maxDocs=43254)
                0.008717075 = queryNorm
              0.3821352 = fieldWeight in 5882, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.1141634 = idf(docFreq=259, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.025849843 = weight(abstract_txt:studies in 5882) [ClassicSimilarity], result of:
            0.025849843 = score(doc=5882,freq=2.0), product of:
              0.06800998 = queryWeight, product of:
                1.8143103 = boost
                4.300216 = idf(docFreq=1594, maxDocs=43254)
                0.008717075 = queryNorm
              0.38008898 = fieldWeight in 5882, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.300216 = idf(docFreq=1594, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.06991055 = weight(abstract_txt:searchers in 5882) [ClassicSimilarity], result of:
            0.06991055 = score(doc=5882,freq=2.0), product of:
              0.1320168 = queryWeight, product of:
                2.5277834 = boost
                5.9912653 = idf(docFreq=293, maxDocs=43254)
                0.008717075 = queryNorm
              0.529558 = fieldWeight in 5882, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.9912653 = idf(docFreq=293, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.057644658 = weight(abstract_txt:query in 5882) [ClassicSimilarity], result of:
            0.057644658 = score(doc=5882,freq=2.0), product of:
              0.13763313 = queryWeight, product of:
                3.332047 = boost
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.008717075 = queryNorm
              0.41882837 = fieldWeight in 5882, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
          0.15027891 = weight(abstract_txt:searcher's in 5882) [ClassicSimilarity], result of:
            0.15027891 = score(doc=5882,freq=1.0), product of:
              0.27704015 = queryWeight, product of:
                3.6618168 = boost
                8.679112 = idf(docFreq=19, maxDocs=43254)
                0.008717075 = queryNorm
              0.5424445 = fieldWeight in 5882, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.679112 = idf(docFreq=19, maxDocs=43254)
                0.0625 = fieldNorm(doc=5882)
        0.28 = coord(7/25)
    
  5. Lee, W.M.; Sanderson, M.: Analyzing URL queries (2010) 0.09
    0.09408467 = sum of:
      0.09408467 = product of:
        0.5880292 = sum of:
          0.008254936 = weight(abstract_txt:that in 570) [ClassicSimilarity], result of:
            0.008254936 = score(doc=570,freq=7.0), product of:
              0.02092765 = queryWeight, product of:
                1.0064344 = boost
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.008717075 = queryNorm
              0.39445114 = fieldWeight in 570, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                2.3854163 = idf(docFreq=10822, maxDocs=43254)
                0.0625 = fieldNorm(doc=570)
          0.009792176 = weight(abstract_txt:classification in 570) [ClassicSimilarity], result of:
            0.009792176 = score(doc=570,freq=1.0), product of:
              0.0391892 = queryWeight, product of:
                1.1245087 = boost
                3.9979079 = idf(docFreq=2157, maxDocs=43254)
                0.008717075 = queryNorm
              0.24986924 = fieldWeight in 570, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.9979079 = idf(docFreq=2157, maxDocs=43254)
                0.0625 = fieldNorm(doc=570)
          0.08152186 = weight(abstract_txt:query in 570) [ClassicSimilarity], result of:
            0.08152186 = score(doc=570,freq=4.0), product of:
              0.13763313 = queryWeight, product of:
                3.332047 = boost
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.008717075 = queryNorm
              0.59231275 = fieldWeight in 570, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.738502 = idf(docFreq=1028, maxDocs=43254)
                0.0625 = fieldNorm(doc=570)
          0.48846024 = weight(abstract_txt:intent in 570) [ClassicSimilarity], result of:
            0.48846024 = score(doc=570,freq=2.0), product of:
              0.72074187 = queryWeight, product of:
                10.78337 = boost
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.008717075 = queryNorm
              0.6777187 = fieldWeight in 570, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                7.667512 = idf(docFreq=54, maxDocs=43254)
                0.0625 = fieldNorm(doc=570)
        0.16 = coord(4/25)