Document (#36233)

Author

Osman, D.J.
Yearwood, J.
Vamplew, P.

Title

Automated opinion detection : implications of the level of agreement between human raters

Source

Information processing and management. 46(2010) no.3, S.331-342

Year

2010

Abstract

The ability to agree with the TREC Blog06 opinion assessments was measured for seven human assessors and compared with the submitted results of the Blog06 participants. The assessors achieved a fair level of agreement between their assessments, although the range between the assessors was large. It is recommended that multiple assessors are used to assess opinion data, or a pre-test of assessors is completed to remove the most dissenting assessors from a pool of assessors prior to the assessment process. The possibility of inconsistent assessments in a corpus also raises concerns about training data for an automated opinion detection system (AODS), so a further recommendation is that AODS training data be assembled from a variety of sources. This paper establishes an aspirational value for an AODS by determining the level of agreement achievable by human assessors when assessing the existence of an opinion on a given topic. Knowing the level of agreement amongst humans is important because it sets an upper bound on the expected performance of AODS. While the AODSs surveyed achieved satisfactory results, none achieved a result close to the upper bound.

Similar documents (content)

1. Verberne, S.; Heijden, M. van der; Hinne, M.; Sappelli, M.; Koldijk, S.; Hoenkamp, E.; Kraaij, W.: Reliability and validity of query intent assessments (2013) 0.16

   0.15757427 = sum of:
     0.15757427 = product of:
       0.98483914 = sum of:
         0.008945314 = weight(abstract_txt:data in 1104) [ClassicSimilarity], result of:
           0.008945314 = score(doc=1104,freq=1.0), product of:
             0.04289871 = queryWeight, product of:
               1.372951 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.009365218 = queryNorm
             0.20852174 = fieldWeight in 1104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.014151374 = weight(abstract_txt:between in 1104) [ClassicSimilarity], result of:
           0.014151374 = score(doc=1104,freq=2.0), product of:
             0.046227768 = queryWeight, product of:
               1.4252281 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.009365218 = queryNorm
             0.3061228 = fieldWeight in 1104, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.21580987 = weight(abstract_txt:agreement in 1104) [ClassicSimilarity], result of:
           0.21580987 = score(doc=1104,freq=4.0), product of:
             0.24834436 = queryWeight, product of:
               3.814428 = boost
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.009365218 = queryNorm
             0.8689944 = fieldWeight in 1104, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.7459326 = weight(abstract_txt:assessors in 1104) [ClassicSimilarity], result of:
           0.7459326 = score(doc=1104,freq=3.0), product of:
             0.7872803 = queryWeight, product of:
               9.60466 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.009365218 = queryNorm
             0.94748026 = fieldWeight in 1104, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
       0.16 = coord(4/25)
   

2. Ravana, S.D.; Rajagopal, P.; Balakrishnan, V.: Ranking retrieval systems using pseudo relevance judgments (2015) 0.14

   0.13723761 = sum of:
     0.13723761 = product of:
       0.85773504 = sum of:
         0.040851705 = weight(abstract_txt:pool in 2591) [ClassicSimilarity], result of:
           0.040851705 = score(doc=2591,freq=1.0), product of:
             0.08187417 = queryWeight, product of:
               1.0950798 = boost
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.009365218 = queryNorm
             0.4989572 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.983315 = idf(docFreq=40, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.010006532 = weight(abstract_txt:between in 2591) [ClassicSimilarity], result of:
           0.010006532 = score(doc=2591,freq=1.0), product of:
             0.046227768 = queryWeight, product of:
               1.4252281 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.009365218 = queryNorm
             0.21646151 = fieldWeight in 2591, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.060944248 = weight(abstract_txt:human in 2591) [ClassicSimilarity], result of:
           0.060944248 = score(doc=2591,freq=6.0), product of:
             0.0848435 = queryWeight, product of:
               1.9308219 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.009365218 = queryNorm
             0.7183137 = fieldWeight in 2591, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
         0.7459326 = weight(abstract_txt:assessors in 2591) [ClassicSimilarity], result of:
           0.7459326 = score(doc=2591,freq=3.0), product of:
             0.7872803 = queryWeight, product of:
               9.60466 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.009365218 = queryNorm
             0.94748026 = fieldWeight in 2591, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=2591)
       0.16 = coord(4/25)
   

3. Otterbacher, J.; Radev, D.: Exploring fact-focused relevance and novelty detection (2008) 0.09

   0.09399509 = sum of:
     0.09399509 = product of:
       0.3916462 = sum of:
         0.014151374 = weight(abstract_txt:between in 2210) [ClassicSimilarity], result of:
           0.014151374 = score(doc=2210,freq=2.0), product of:
             0.046227768 = queryWeight, product of:
               1.4252281 = boost
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.009365218 = queryNorm
             0.3061228 = fieldWeight in 2210, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.4633842 = idf(docFreq=3764, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
         0.028250491 = weight(abstract_txt:automated in 2210) [ClassicSimilarity], result of:
           0.028250491 = score(doc=2210,freq=1.0), product of:
             0.080667906 = queryWeight, product of:
               1.5372258 = boost
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.009365218 = queryNorm
             0.35020733 = fieldWeight in 2210, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.6033173 = idf(docFreq=442, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
         0.08684665 = weight(abstract_txt:detection in 2210) [ClassicSimilarity], result of:
           0.08684665 = score(doc=2210,freq=3.0), product of:
             0.11825289 = queryWeight, product of:
               1.8612006 = boost
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.009365218 = queryNorm
             0.73441464 = fieldWeight in 2210, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.784232 = idf(docFreq=135, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
         0.024880385 = weight(abstract_txt:human in 2210) [ClassicSimilarity], result of:
           0.024880385 = score(doc=2210,freq=1.0), product of:
             0.0848435 = queryWeight, product of:
               1.9308219 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.009365218 = queryNorm
             0.29325032 = fieldWeight in 2210, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
         0.05062051 = weight(abstract_txt:level in 2210) [ClassicSimilarity], result of:
           0.05062051 = score(doc=2210,freq=3.0), product of:
             0.10396106 = queryWeight, product of:
               2.4679573 = boost
               4.497956 = idf(docFreq=1337, maxDocs=44218)
               0.009365218 = queryNorm
             0.486918 = fieldWeight in 2210, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.497956 = idf(docFreq=1337, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
         0.18689682 = weight(abstract_txt:agreement in 2210) [ClassicSimilarity], result of:
           0.18689682 = score(doc=2210,freq=3.0), product of:
             0.24834436 = queryWeight, product of:
               3.814428 = boost
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.009365218 = queryNorm
             0.7525712 = fieldWeight in 2210, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.9519553 = idf(docFreq=114, maxDocs=44218)
               0.0625 = fieldNorm(doc=2210)
       0.24 = coord(6/25)
   

4. Chu, H.: Factors affecting relevance judgment : a report from TREC Legal track (2011) 0.09

   0.09357099 = sum of:
     0.09357099 = product of:
       0.7797583 = sum of:
         0.008945314 = weight(abstract_txt:data in 4540) [ClassicSimilarity], result of:
           0.008945314 = score(doc=4540,freq=1.0), product of:
             0.04289871 = queryWeight, product of:
               1.372951 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.009365218 = queryNorm
             0.20852174 = fieldWeight in 4540, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.024880385 = weight(abstract_txt:human in 4540) [ClassicSimilarity], result of:
           0.024880385 = score(doc=4540,freq=1.0), product of:
             0.0848435 = queryWeight, product of:
               1.9308219 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.009365218 = queryNorm
             0.29325032 = fieldWeight in 4540, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
         0.7459326 = weight(abstract_txt:assessors in 4540) [ClassicSimilarity], result of:
           0.7459326 = score(doc=4540,freq=3.0), product of:
             0.7872803 = queryWeight, product of:
               9.60466 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.009365218 = queryNorm
             0.94748026 = fieldWeight in 4540, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=4540)
       0.12 = coord(3/25)
   

5. Ruthven, I.: Relevance behaviour in TREC (2014) 0.09

   0.08983289 = sum of:
     0.08983289 = product of:
       0.56145555 = sum of:
         0.02000233 = weight(abstract_txt:data in 1785) [ClassicSimilarity], result of:
           0.02000233 = score(doc=1785,freq=5.0), product of:
             0.04289871 = queryWeight, product of:
               1.372951 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.009365218 = queryNorm
             0.46626878 = fieldWeight in 1785, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.024880385 = weight(abstract_txt:human in 1785) [ClassicSimilarity], result of:
           0.024880385 = score(doc=1785,freq=1.0), product of:
             0.0848435 = queryWeight, product of:
               1.9308219 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.009365218 = queryNorm
             0.29325032 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.08590844 = weight(abstract_txt:assessments in 1785) [ClassicSimilarity], result of:
           0.08590844 = score(doc=1785,freq=1.0), product of:
             0.1938226 = queryWeight, product of:
               2.918335 = boost
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.009365218 = queryNorm
             0.44323233 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0917172 = idf(docFreq=99, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
         0.4306644 = weight(abstract_txt:assessors in 1785) [ClassicSimilarity], result of:
           0.4306644 = score(doc=1785,freq=1.0), product of:
             0.7872803 = queryWeight, product of:
               9.60466 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.009365218 = queryNorm
             0.547028 = fieldWeight in 1785, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=1785)
       0.16 = coord(4/25)