Document (#42584)

Author
Saracevic, T.
Title
Effects of inconsistent relevance judgments on information retrieval test results : a historical perspective
Source
Library trends. 56(2008) no.4, S.763-783
Year
2008
Abstract
The main objective of information retrieval (IR) systems is to retrieve information or information objects relevant to user requests and possible needs. In IR tests, retrieval effectiveness is established by comparing IR systems retrievals (systems relevance) with users' or user surrogates' assessments (user relevance), where user relevance is treated as the gold standard for performance evaluation. Relevance is a human notion, and establishing relevance by humans is fraught with a number of problems-inconsistency in judgment being one of them. The aim of this critical review is to explore the relationship between relevance on the one hand and testing of IR systems and procedures on the other. Critics of IR tests raised the issue of validity of the IR tests because they were based on relevance judgments that are inconsistent. This review traces and synthesizes experimental studies dealing with (1) inconsistency of relevance judgments by people, (2) effects of such inconsistency on results of IR tests and (3) reasons for retrieval failures. A historical context for these studies and for IR testing is provided including an assessment of Lancaster's (1969) evaluation of MEDLARS and its unique place in the history of IR evaluation.
Content
Beitrag in einem Themenheft 'The Influence of F. W. Lancaster on Information Science and on Libraries', das als Festschrift für F.W. Lancaster deklariert ist.
Footnote
Vgl.: 10.1353/lib.0.0000.
Theme
Retrievalstudien

Similar documents (author)

  1. Saracevic, T.: ¬The concept of 'relevance' in information science : a historical review (1970) 5.00
    4.995577 = sum of:
      4.995577 = weight(author_txt:saracevic in 2010) [ClassicSimilarity], result of:
        4.995577 = fieldWeight in 2010, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          7.9929233 = idf(docFreq=39, maxDocs=43556)
          0.625 = fieldNorm(doc=2010)
    
  2. Saracevic, T.: Information retrieval (1985) 5.00
    4.995577 = sum of:
      4.995577 = weight(author_txt:saracevic in 3302) [ClassicSimilarity], result of:
        4.995577 = fieldWeight in 3302, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          7.9929233 = idf(docFreq=39, maxDocs=43556)
          0.625 = fieldNorm(doc=3302)
    
  3. Saracevic, T.: Individual differences in organizing, searching and retrieving information (1991) 5.00
    4.995577 = sum of:
      4.995577 = weight(author_txt:saracevic in 3692) [ClassicSimilarity], result of:
        4.995577 = fieldWeight in 3692, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          7.9929233 = idf(docFreq=39, maxDocs=43556)
          0.625 = fieldNorm(doc=3692)
    
  4. Saracevic, T.: Charting the future of information science (1990) 5.00
    4.995577 = sum of:
      4.995577 = weight(author_txt:saracevic in 4078) [ClassicSimilarity], result of:
        4.995577 = fieldWeight in 4078, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          7.9929233 = idf(docFreq=39, maxDocs=43556)
          0.625 = fieldNorm(doc=4078)
    
  5. Saracevic, T: Ten years of relevance experimentation : a summary and sythesis of conclusions (1970) 5.00
    4.995577 = sum of:
      4.995577 = weight(author_txt:saracevic in 5356) [ClassicSimilarity], result of:
        4.995577 = fieldWeight in 5356, product of:
          1.0 = tf(freq=1.0), with freq of:
            1.0 = termFreq=1.0
          7.9929233 = idf(docFreq=39, maxDocs=43556)
          0.625 = fieldNorm(doc=5356)
    

Similar documents (content)

  1. Spink, A.; Greisdorf, H.: Regions and levels : Measuring and mapping users' relevance judgements (2001) 0.28
    0.27515027 = sum of:
      0.27515027 = product of:
        0.9826795 = sum of:
          0.020695454 = weight(abstract_txt:studies in 584) [ClassicSimilarity], result of:
            0.020695454 = score(doc=584,freq=1.0), product of:
              0.07718996 = queryWeight, product of:
                1.1399968 = boost
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.015784219 = queryNorm
              0.2681107 = fieldWeight in 584, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.01057756 = weight(abstract_txt:information in 584) [ClassicSimilarity], result of:
            0.01057756 = score(doc=584,freq=2.0), product of:
              0.04934402 = queryWeight, product of:
                1.2890077 = boost
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.015784219 = queryNorm
              0.21436356 = fieldWeight in 584, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.050454658 = weight(abstract_txt:evaluation in 584) [ClassicSimilarity], result of:
            0.050454658 = score(doc=584,freq=2.0), product of:
              0.12703755 = queryWeight, product of:
                1.7911621 = boost
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.015784219 = queryNorm
              0.39716333 = fieldWeight in 584, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.02954835 = weight(abstract_txt:systems in 584) [ClassicSimilarity], result of:
            0.02954835 = score(doc=584,freq=2.0), product of:
              0.09787378 = queryWeight, product of:
                1.8153961 = boost
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.015784219 = queryNorm
              0.30190262 = fieldWeight in 584, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.02198052 = weight(abstract_txt:retrieval in 584) [ClassicSimilarity], result of:
            0.02198052 = score(doc=584,freq=1.0), product of:
              0.10123861 = queryWeight, product of:
                1.8463384 = boost
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.015784219 = queryNorm
              0.21711598 = fieldWeight in 584, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.3191449 = weight(abstract_txt:judgments in 584) [ClassicSimilarity], result of:
            0.3191449 = score(doc=584,freq=6.0), product of:
              0.30126554 = queryWeight, product of:
                2.7583132 = boost
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.015784219 = queryNorm
              1.0593475 = fieldWeight in 584, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
          0.530278 = weight(abstract_txt:relevance in 584) [ClassicSimilarity], result of:
            0.530278 = score(doc=584,freq=14.0), product of:
              0.45955917 = queryWeight, product of:
                5.900655 = boost
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.015784219 = queryNorm
              1.1538842 = fieldWeight in 584, product of:
                3.7416575 = tf(freq=14.0), with freq of:
                  14.0 = termFreq=14.0
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.0625 = fieldNorm(doc=584)
        0.28 = coord(7/25)
    
  2. Harter, S.P.: Variations in relevance assessments and the measurement of retrieval effectiveness (1996) 0.25
    0.25152296 = sum of:
      0.25152296 = product of:
        0.69867486 = sum of:
          0.01057756 = weight(abstract_txt:information in 3070) [ClassicSimilarity], result of:
            0.01057756 = score(doc=3070,freq=2.0), product of:
              0.04934402 = queryWeight, product of:
                1.2890077 = boost
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.015784219 = queryNorm
              0.21436356 = fieldWeight in 3070, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.030393807 = weight(abstract_txt:review in 3070) [ClassicSimilarity], result of:
            0.030393807 = score(doc=3070,freq=1.0), product of:
              0.09973194 = queryWeight, product of:
                1.2958071 = boost
                4.87608 = idf(docFreq=902, maxDocs=43556)
                0.015784219 = queryNorm
              0.304755 = fieldWeight in 3070, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.87608 = idf(docFreq=902, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.06541658 = weight(abstract_txt:effects in 3070) [ClassicSimilarity], result of:
            0.06541658 = score(doc=3070,freq=2.0), product of:
              0.13195507 = queryWeight, product of:
                1.4905146 = boost
                5.6087584 = idf(docFreq=433, maxDocs=43556)
                0.015784219 = queryNorm
              0.49574888 = fieldWeight in 3070, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.6087584 = idf(docFreq=433, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.06179408 = weight(abstract_txt:evaluation in 3070) [ClassicSimilarity], result of:
            0.06179408 = score(doc=3070,freq=3.0), product of:
              0.12703755 = queryWeight, product of:
                1.7911621 = boost
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.015784219 = queryNorm
              0.48642373 = fieldWeight in 3070, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.020893838 = weight(abstract_txt:systems in 3070) [ClassicSimilarity], result of:
            0.020893838 = score(doc=3070,freq=1.0), product of:
              0.09787378 = queryWeight, product of:
                1.8153961 = boost
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.015784219 = queryNorm
              0.21347739 = fieldWeight in 3070, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.049149938 = weight(abstract_txt:retrieval in 3070) [ClassicSimilarity], result of:
            0.049149938 = score(doc=3070,freq=5.0), product of:
              0.10123861 = queryWeight, product of:
                1.8463384 = boost
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.015784219 = queryNorm
              0.4854861 = fieldWeight in 3070, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.02614708 = weight(abstract_txt:user in 3070) [ClassicSimilarity], result of:
            0.02614708 = score(doc=3070,freq=1.0), product of:
              0.113658786 = queryWeight, product of:
                1.9563192 = boost
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.015784219 = queryNorm
              0.23004892 = fieldWeight in 3070, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.11740023 = weight(abstract_txt:tests in 3070) [ClassicSimilarity], result of:
            0.11740023 = score(doc=3070,freq=1.0), product of:
              0.30933815 = queryWeight, product of:
                3.227416 = boost
                6.072331 = idf(docFreq=272, maxDocs=43556)
                0.015784219 = queryNorm
              0.37952068 = fieldWeight in 3070, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.072331 = idf(docFreq=272, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
          0.31690174 = weight(abstract_txt:relevance in 3070) [ClassicSimilarity], result of:
            0.31690174 = score(doc=3070,freq=5.0), product of:
              0.45955917 = queryWeight, product of:
                5.900655 = boost
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.015784219 = queryNorm
              0.68957764 = fieldWeight in 3070, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.0625 = fieldNorm(doc=3070)
        0.36 = coord(9/25)
    
  3. Bateman, J.: Modelling the importance of end-user relevance criteria (1999) 0.24
    0.23910274 = sum of:
      0.23910274 = product of:
        0.7471961 = sum of:
          0.0054323734 = weight(abstract_txt:with in 604) [ClassicSimilarity], result of:
            0.0054323734 = score(doc=604,freq=1.0), product of:
              0.039597 = queryWeight, product of:
                2.508645 = idf(docFreq=9634, maxDocs=43556)
                0.015784219 = queryNorm
              0.13719153 = fieldWeight in 604, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.508645 = idf(docFreq=9634, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.01810852 = weight(abstract_txt:studies in 604) [ClassicSimilarity], result of:
            0.01810852 = score(doc=604,freq=1.0), product of:
              0.07718996 = queryWeight, product of:
                1.1399968 = boost
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.015784219 = queryNorm
              0.23459685 = fieldWeight in 604, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.020695627 = weight(abstract_txt:information in 604) [ClassicSimilarity], result of:
            0.020695627 = score(doc=604,freq=10.0), product of:
              0.04934402 = queryWeight, product of:
                1.2890077 = boost
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.015784219 = queryNorm
              0.4194151 = fieldWeight in 604, product of:
                3.1622777 = tf(freq=10.0), with freq of:
                  10.0 = termFreq=10.0
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.031217229 = weight(abstract_txt:evaluation in 604) [ClassicSimilarity], result of:
            0.031217229 = score(doc=604,freq=1.0), product of:
              0.12703755 = queryWeight, product of:
                1.7911621 = boost
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.015784219 = queryNorm
              0.24573228 = fieldWeight in 604, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.019232955 = weight(abstract_txt:retrieval in 604) [ClassicSimilarity], result of:
            0.019232955 = score(doc=604,freq=1.0), product of:
              0.10123861 = queryWeight, product of:
                1.8463384 = boost
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.015784219 = queryNorm
              0.18997648 = fieldWeight in 604, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.03235536 = weight(abstract_txt:user in 604) [ClassicSimilarity], result of:
            0.03235536 = score(doc=604,freq=2.0), product of:
              0.113658786 = queryWeight, product of:
                1.9563192 = boost
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.015784219 = queryNorm
              0.284671 = fieldWeight in 604, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.22800812 = weight(abstract_txt:judgments in 604) [ClassicSimilarity], result of:
            0.22800812 = score(doc=604,freq=4.0), product of:
              0.30126554 = queryWeight, product of:
                2.7583132 = boost
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.015784219 = queryNorm
              0.7568344 = fieldWeight in 604, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
          0.39214587 = weight(abstract_txt:relevance in 604) [ClassicSimilarity], result of:
            0.39214587 = score(doc=604,freq=10.0), product of:
              0.45955917 = queryWeight, product of:
                5.900655 = boost
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.015784219 = queryNorm
              0.8533088 = fieldWeight in 604, product of:
                3.1622777 = tf(freq=10.0), with freq of:
                  10.0 = termFreq=10.0
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.0546875 = fieldNorm(doc=604)
        0.32 = coord(8/25)
    
  4. Choi, Y.; Rasmussen, E.M.: Users' relevance criteria in image retrieval in American history (2002) 0.23
    0.23144493 = sum of:
      0.23144493 = product of:
        0.7232654 = sum of:
          0.01810852 = weight(abstract_txt:studies in 3590) [ClassicSimilarity], result of:
            0.01810852 = score(doc=3590,freq=1.0), product of:
              0.07718996 = queryWeight, product of:
                1.1399968 = boost
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.015784219 = queryNorm
              0.23459685 = fieldWeight in 3590, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.013089064 = weight(abstract_txt:information in 3590) [ClassicSimilarity], result of:
            0.013089064 = score(doc=3590,freq=4.0), product of:
              0.04934402 = queryWeight, product of:
                1.2890077 = boost
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.015784219 = queryNorm
              0.26526138 = fieldWeight in 3590, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.031217229 = weight(abstract_txt:evaluation in 3590) [ClassicSimilarity], result of:
            0.031217229 = score(doc=3590,freq=1.0), product of:
              0.12703755 = queryWeight, product of:
                1.7911621 = boost
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.015784219 = queryNorm
              0.24573228 = fieldWeight in 3590, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.49339 = idf(docFreq=1323, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.018282108 = weight(abstract_txt:systems in 3590) [ClassicSimilarity], result of:
            0.018282108 = score(doc=3590,freq=1.0), product of:
              0.09787378 = queryWeight, product of:
                1.8153961 = boost
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.015784219 = queryNorm
              0.18679272 = fieldWeight in 3590, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.027199505 = weight(abstract_txt:retrieval in 3590) [ClassicSimilarity], result of:
            0.027199505 = score(doc=3590,freq=2.0), product of:
              0.10123861 = queryWeight, product of:
                1.8463384 = boost
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.015784219 = queryNorm
              0.2686673 = fieldWeight in 3590, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.03235536 = weight(abstract_txt:user in 3590) [ClassicSimilarity], result of:
            0.03235536 = score(doc=3590,freq=2.0), product of:
              0.113658786 = queryWeight, product of:
                1.9563192 = boost
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.015784219 = queryNorm
              0.284671 = fieldWeight in 3590, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.2549208 = weight(abstract_txt:judgments in 3590) [ClassicSimilarity], result of:
            0.2549208 = score(doc=3590,freq=5.0), product of:
              0.30126554 = queryWeight, product of:
                2.7583132 = boost
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.015784219 = queryNorm
              0.84616655 = fieldWeight in 3590, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
          0.32809278 = weight(abstract_txt:relevance in 3590) [ClassicSimilarity], result of:
            0.32809278 = score(doc=3590,freq=7.0), product of:
              0.45955917 = queryWeight, product of:
                5.900655 = boost
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.015784219 = queryNorm
              0.71392936 = fieldWeight in 3590, product of:
                2.6457512 = tf(freq=7.0), with freq of:
                  7.0 = termFreq=7.0
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.0546875 = fieldNorm(doc=3590)
        0.32 = coord(8/25)
    
  5. Taylor, A.: User relevance criteria choices and the information search process (2012) 0.23
    0.22647655 = sum of:
      0.22647655 = product of:
        0.8088448 = sum of:
          0.02926779 = weight(abstract_txt:studies in 4743) [ClassicSimilarity], result of:
            0.02926779 = score(doc=4743,freq=2.0), product of:
              0.07718996 = queryWeight, product of:
                1.1399968 = boost
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.015784219 = queryNorm
              0.37916577 = fieldWeight in 4743, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.289771 = idf(docFreq=1622, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.02480658 = weight(abstract_txt:information in 4743) [ClassicSimilarity], result of:
            0.02480658 = score(doc=4743,freq=11.0), product of:
              0.04934402 = queryWeight, product of:
                1.2890077 = boost
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.015784219 = queryNorm
              0.50272715 = fieldWeight in 4743, product of:
                3.3166249 = tf(freq=11.0), with freq of:
                  11.0 = termFreq=11.0
                2.425247 = idf(docFreq=10472, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.020893838 = weight(abstract_txt:systems in 4743) [ClassicSimilarity], result of:
            0.020893838 = score(doc=4743,freq=1.0), product of:
              0.09787378 = queryWeight, product of:
                1.8153961 = boost
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.015784219 = queryNorm
              0.21347739 = fieldWeight in 4743, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4156382 = idf(docFreq=3889, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.02198052 = weight(abstract_txt:retrieval in 4743) [ClassicSimilarity], result of:
            0.02198052 = score(doc=4743,freq=1.0), product of:
              0.10123861 = queryWeight, product of:
                1.8463384 = boost
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.015784219 = queryNorm
              0.21711598 = fieldWeight in 4743, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.4738557 = idf(docFreq=3669, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.02614708 = weight(abstract_txt:user in 4743) [ClassicSimilarity], result of:
            0.02614708 = score(doc=4743,freq=1.0), product of:
              0.113658786 = queryWeight, product of:
                1.9563192 = boost
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.015784219 = queryNorm
              0.23004892 = fieldWeight in 4743, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                3.6807828 = idf(docFreq=2983, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.26058072 = weight(abstract_txt:judgments in 4743) [ClassicSimilarity], result of:
            0.26058072 = score(doc=4743,freq=4.0), product of:
              0.30126554 = queryWeight, product of:
                2.7583132 = boost
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.015784219 = queryNorm
              0.8649536 = fieldWeight in 4743, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                6.9196286 = idf(docFreq=116, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
          0.4251683 = weight(abstract_txt:relevance in 4743) [ClassicSimilarity], result of:
            0.4251683 = score(doc=4743,freq=9.0), product of:
              0.45955917 = queryWeight, product of:
                5.900655 = boost
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.015784219 = queryNorm
              0.92516553 = fieldWeight in 4743, product of:
                3.0 = tf(freq=9.0), with freq of:
                  9.0 = termFreq=9.0
                4.934216 = idf(docFreq=851, maxDocs=43556)
                0.0625 = fieldNorm(doc=4743)
        0.28 = coord(7/25)