Document (#44072)

Author

Berg, A.
Nelimarkka, M.

Title

Do you see what I see? : measuring the semantic differences in image-recognition services' outputs

Source

Journal of the Association for Information Science and Technology. 74(2023) no.11, S.1307-1324

Year

2023

Abstract

As scholars increasingly undertake large-scale analysis of visual materials, advanced computational tools show promise for informing that process. One technique in the toolbox is image recognition, made readily accessible via Google Vision AI, Microsoft Azure Computer Vision, and Amazon's Rekognition service. However, concerns about such issues as bias factors and low reliability have led to warnings against research employing it. A systematic study of cross-service label agreement concretized such issues: using eight datasets, spanning professionally produced and user-generated images, the work showed that image-recognition services disagree on the most suitable labels for images. Beyond supporting caveats expressed in prior literature, the report articulates two mitigation strategies, both involving the use of multiple image-recognition services: Highly explorative research could include all the labels, accepting noisier but less restrictive analysis output. Alternatively, scholars may employ word-embedding-based approaches to identify concepts that are similar enough for their purposes, then focus on those labels filtered in.

Content

Vgl.: https://asistdl.onlinelibrary.wiley.com/toc/23301643/current. https://doi.org/10.1002/asi.24827.

Field

Kognitionswissenschaft
Informatik

Form

Bilder

Similar documents (author)

1. Berg, O.: Current problems with MARC/ISBD formats in relation to online public access of bibliographic information (1991) 5.41

   5.4077277 = sum of:
     5.4077277 = weight(author_txt:berg in 469) [ClassicSimilarity], result of:
       5.4077277 = score(doc=469,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.115575336 = queryNorm
         5.407728 = fieldWeight in 469, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.625 = fieldNorm(doc=469)
   

2. Berg, S.: Auf dem Weg : Fallbeispiel: Vorbereitungen für einen elektronischen Katalog (1995) 5.41

   5.4077277 = sum of:
     5.4077277 = weight(author_txt:berg in 648) [ClassicSimilarity], result of:
       5.4077277 = score(doc=648,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.115575336 = queryNorm
         5.407728 = fieldWeight in 648, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.625 = fieldNorm(doc=648)
   

3. Berg, L.: Wie das Internet die Gesellschaft verändert : Google gründet ein Forschungsinstitut in Berlin (2011) 5.41

   5.4077277 = sum of:
     5.4077277 = weight(author_txt:berg in 4552) [ClassicSimilarity], result of:
       5.4077277 = score(doc=4552,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.115575336 = queryNorm
         5.407728 = fieldWeight in 4552, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.625 = fieldNorm(doc=4552)
   

4. Berg, L.: Pablo will es wissen : Lernen mit Salman Khan (2012) 5.41

   5.4077277 = sum of:
     5.4077277 = weight(author_txt:berg in 228) [ClassicSimilarity], result of:
       5.4077277 = score(doc=228,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.115575336 = queryNorm
         5.407728 = fieldWeight in 228, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.625 = fieldNorm(doc=228)
   

5. Berg, J. van den: ¬The ICONCLASS browser user's guide (1992) 4.33

   4.326182 = sum of:
     4.326182 = weight(author_txt:berg in 3270) [ClassicSimilarity], result of:
       4.326182 = score(doc=3270,freq=1.0), product of:
         0.99999994 = queryWeight, product of:
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.115575336 = queryNorm
         4.3261824 = fieldWeight in 3270, product of:
           1.0 = tf(freq=1.0), with freq of:
             1.0 = termFreq=1.0
           8.652365 = idf(docFreq=20, maxDocs=44218)
           0.5 = fieldNorm(doc=3270)
   

Similar documents (content)

1. Heidorn, P.B.: Image retrieval as linguistic and nonlinguistic visual model matching (1999) 0.14

   0.13587366 = sum of:
     0.13587366 = product of:
       0.67936826 = sum of:
         0.06830381 = weight(abstract_txt:images in 841) [ClassicSimilarity], result of:
           0.06830381 = score(doc=841,freq=2.0), product of:
             0.14237271 = queryWeight, product of:
               1.4840449 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.017674884 = queryNorm
             0.4797535 = fieldWeight in 841, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.0625 = fieldNorm(doc=841)
         0.078939304 = weight(abstract_txt:vision in 841) [ClassicSimilarity], result of:
           0.078939304 = score(doc=841,freq=1.0), product of:
             0.19754635 = queryWeight, product of:
               1.7481077 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.017674884 = queryNorm
             0.3995989 = fieldWeight in 841, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.0625 = fieldNorm(doc=841)
         0.16241352 = weight(abstract_txt:image in 841) [ClassicSimilarity], result of:
           0.16241352 = score(doc=841,freq=3.0), product of:
             0.279163 = queryWeight, product of:
               2.938851 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.017674884 = queryNorm
             0.5817874 = fieldWeight in 841, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.0625 = fieldNorm(doc=841)
         0.17380032 = weight(abstract_txt:labels in 841) [ClassicSimilarity], result of:
           0.17380032 = score(doc=841,freq=1.0), product of:
             0.382711 = queryWeight, product of:
               2.9799898 = boost
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.017674884 = queryNorm
             0.4541294 = fieldWeight in 841, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.0625 = fieldNorm(doc=841)
         0.1959113 = weight(abstract_txt:recognition in 841) [ClassicSimilarity], result of:
           0.1959113 = score(doc=841,freq=2.0), product of:
             0.36211497 = queryWeight, product of:
               3.347125 = boost
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.017674884 = queryNorm
             0.5410196 = fieldWeight in 841, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.0625 = fieldNorm(doc=841)
       0.2 = coord(5/25)
   

2. Berinstein, P.: Images in your future : the missing picture in an online search (1997) 0.10

   0.10146489 = sum of:
     0.10146489 = product of:
       0.6341556 = sum of:
         0.16904329 = weight(abstract_txt:images in 556) [ClassicSimilarity], result of:
           0.16904329 = score(doc=556,freq=4.0), product of:
             0.14237271 = queryWeight, product of:
               1.4840449 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.017674884 = queryNorm
             1.1873293 = fieldWeight in 556, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.109375 = fieldNorm(doc=556)
         0.058587816 = weight(abstract_txt:services in 556) [ClassicSimilarity], result of:
           0.058587816 = score(doc=556,freq=1.0), product of:
             0.1276488 = queryWeight, product of:
               1.721027 = boost
               4.1963577 = idf(docFreq=1808, maxDocs=44218)
               0.017674884 = queryNorm
             0.45897663 = fieldWeight in 556, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1963577 = idf(docFreq=1808, maxDocs=44218)
               0.109375 = fieldNorm(doc=556)
         0.16409661 = weight(abstract_txt:image in 556) [ClassicSimilarity], result of:
           0.16409661 = score(doc=556,freq=1.0), product of:
             0.279163 = queryWeight, product of:
               2.938851 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.017674884 = queryNorm
             0.5878165 = fieldWeight in 556, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.109375 = fieldNorm(doc=556)
         0.24242787 = weight(abstract_txt:recognition in 556) [ClassicSimilarity], result of:
           0.24242787 = score(doc=556,freq=1.0), product of:
             0.36211497 = queryWeight, product of:
               3.347125 = boost
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.017674884 = queryNorm
             0.66947764 = fieldWeight in 556, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.109375 = fieldNorm(doc=556)
       0.16 = coord(4/25)
   

3. Heidorn, P.B.; Wei, Q.: Automatic metadata extraction from museum specimen labels (2008) 0.08

   0.082679644 = sum of:
     0.082679644 = product of:
       0.5167478 = sum of:
         0.028888516 = weight(abstract_txt:service in 2624) [ClassicSimilarity], result of:
           0.028888516 = score(doc=2624,freq=1.0), product of:
             0.10107032 = queryWeight, product of:
               1.25039 = boost
               4.5732145 = idf(docFreq=1240, maxDocs=44218)
               0.017674884 = queryNorm
             0.2858259 = fieldWeight in 2624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5732145 = idf(docFreq=1240, maxDocs=44218)
               0.0625 = fieldNorm(doc=2624)
         0.048298083 = weight(abstract_txt:images in 2624) [ClassicSimilarity], result of:
           0.048298083 = score(doc=2624,freq=1.0), product of:
             0.14237271 = queryWeight, product of:
               1.4840449 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.017674884 = queryNorm
             0.33923694 = fieldWeight in 2624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.0625 = fieldNorm(doc=2624)
         0.30103096 = weight(abstract_txt:labels in 2624) [ClassicSimilarity], result of:
           0.30103096 = score(doc=2624,freq=3.0), product of:
             0.382711 = queryWeight, product of:
               2.9799898 = boost
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.017674884 = queryNorm
             0.7865752 = fieldWeight in 2624, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.0625 = fieldNorm(doc=2624)
         0.13853021 = weight(abstract_txt:recognition in 2624) [ClassicSimilarity], result of:
           0.13853021 = score(doc=2624,freq=1.0), product of:
             0.36211497 = queryWeight, product of:
               3.347125 = boost
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.017674884 = queryNorm
             0.38255864 = fieldWeight in 2624, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.0625 = fieldNorm(doc=2624)
       0.16 = coord(4/25)
   

4. Forsyth, D.A.: Computer vision tools for finding images and video sequences (1999) 0.08

   0.08247382 = sum of:
     0.08247382 = product of:
       0.5154614 = sum of:
         0.13091607 = weight(abstract_txt:filtered in 835) [ClassicSimilarity], result of:
           0.13091607 = score(doc=835,freq=1.0), product of:
             0.16764784 = queryWeight, product of:
               1.1387218 = boost
               8.329592 = idf(docFreq=28, maxDocs=44218)
               0.017674884 = queryNorm
             0.7808992 = fieldWeight in 835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.329592 = idf(docFreq=28, maxDocs=44218)
               0.09375 = fieldNorm(doc=835)
         0.1254821 = weight(abstract_txt:images in 835) [ClassicSimilarity], result of:
           0.1254821 = score(doc=835,freq=3.0), product of:
             0.14237271 = queryWeight, product of:
               1.4840449 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.017674884 = queryNorm
             0.8813634 = fieldWeight in 835, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.09375 = fieldNorm(doc=835)
         0.11840896 = weight(abstract_txt:vision in 835) [ClassicSimilarity], result of:
           0.11840896 = score(doc=835,freq=1.0), product of:
             0.19754635 = queryWeight, product of:
               1.7481077 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.017674884 = queryNorm
             0.5993984 = fieldWeight in 835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.09375 = fieldNorm(doc=835)
         0.14065425 = weight(abstract_txt:image in 835) [ClassicSimilarity], result of:
           0.14065425 = score(doc=835,freq=1.0), product of:
             0.279163 = queryWeight, product of:
               2.938851 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.017674884 = queryNorm
             0.5038427 = fieldWeight in 835, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.09375 = fieldNorm(doc=835)
       0.16 = coord(4/25)
   

5. Su, Z.; Li, D.; Li, H.; Luo, X.: Boosting attribute recognition with latent topics by matrix factorization (2017) 0.08

   0.078035474 = sum of:
     0.078035474 = product of:
       0.6502956 = sum of:
         0.060372606 = weight(abstract_txt:images in 3693) [ClassicSimilarity], result of:
           0.060372606 = score(doc=3693,freq=1.0), product of:
             0.14237271 = queryWeight, product of:
               1.4840449 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.017674884 = queryNorm
             0.4240462 = fieldWeight in 3693, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.078125 = fieldNorm(doc=3693)
         0.16576262 = weight(abstract_txt:image in 3693) [ClassicSimilarity], result of:
           0.16576262 = score(doc=3693,freq=2.0), product of:
             0.279163 = queryWeight, product of:
               2.938851 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.017674884 = queryNorm
             0.59378433 = fieldWeight in 3693, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.078125 = fieldNorm(doc=3693)
         0.42416042 = weight(abstract_txt:recognition in 3693) [ClassicSimilarity], result of:
           0.42416042 = score(doc=3693,freq=6.0), product of:
             0.36211497 = queryWeight, product of:
               3.347125 = boost
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.017674884 = queryNorm
             1.1713419 = fieldWeight in 3693, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.1209383 = idf(docFreq=263, maxDocs=44218)
               0.078125 = fieldNorm(doc=3693)
       0.12 = coord(3/25)