Document (#38871)

Author
Vinyals, O.
Toshev, A.
Bengio, S.
Erhan, D.
Title
Show and tell : a neural image caption generator
Source
http://arxiv.org/abs/1411.4555
Year
2014
Abstract
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.
Content
Vgl.: http://googleresearch.blogspot.de/2014/11/a-picture-is-worth-thousand-coherent.html. Vgl. auch: https://news.ycombinator.com/item?id=8621658.
Theme
Automatisches Indexieren
Form
Bilder
Object
Google

Similar documents (content)

  1. Karpathy, A.; Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions (2015) 0.39
    0.3868583 = sum of:
      0.3868583 = product of:
        1.0746064 = sum of:
          0.03149767 = weight(abstract_txt:language in 3333) [ClassicSimilarity], result of:
            0.03149767 = score(doc=3333,freq=1.0), product of:
              0.09617234 = queryWeight, product of:
                1.0465647 = boost
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.021920273 = queryNorm
              0.32751274 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.131129 = weight(abstract_txt:generative in 3333) [ClassicSimilarity], result of:
            0.131129 = score(doc=3333,freq=1.0), product of:
              0.19753942 = queryWeight, product of:
                1.060603 = boost
                8.496791 = idf(docFreq=23, maxDocs=43254)
                0.021920273 = queryNorm
              0.6638118 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.496791 = idf(docFreq=23, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.20895185 = weight(abstract_txt:recurrent in 3333) [ClassicSimilarity], result of:
            0.20895185 = score(doc=3333,freq=2.0), product of:
              0.213899 = queryWeight, product of:
                1.1036475 = boost
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.021920273 = queryNorm
              0.97687155 = fieldWeight in 3333, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.048559926 = weight(abstract_txt:state in 3333) [ClassicSimilarity], result of:
            0.048559926 = score(doc=3333,freq=1.0), product of:
              0.12834628 = queryWeight, product of:
                1.2090176 = boost
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.021920273 = queryNorm
              0.37835088 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.05693799 = weight(abstract_txt:natural in 3333) [ClassicSimilarity], result of:
            0.05693799 = score(doc=3333,freq=1.0), product of:
              0.14271381 = queryWeight, product of:
                1.2748938 = boost
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.021920273 = queryNorm
              0.3989662 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.056211308 = weight(abstract_txt:show in 3333) [ClassicSimilarity], result of:
            0.056211308 = score(doc=3333,freq=1.0), product of:
              0.16197357 = queryWeight, product of:
                1.663446 = boost
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.021920273 = queryNorm
              0.34704 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.14391546 = weight(abstract_txt:dataset in 3333) [ClassicSimilarity], result of:
            0.14391546 = score(doc=3333,freq=1.0), product of:
              0.26481113 = queryWeight, product of:
                1.7366372 = boost
                6.956346 = idf(docFreq=111, maxDocs=43254)
                0.021920273 = queryNorm
              0.54346454 = fieldWeight in 3333, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.956346 = idf(docFreq=111, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.11007918 = weight(abstract_txt:model in 3333) [ClassicSimilarity], result of:
            0.11007918 = score(doc=3333,freq=4.0), product of:
              0.17578888 = queryWeight, product of:
                2.0010211 = boost
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.021920273 = queryNorm
              0.62620103 = fieldWeight in 3333, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
          0.28732398 = weight(abstract_txt:image in 3333) [ClassicSimilarity], result of:
            0.28732398 = score(doc=3333,freq=3.0), product of:
              0.3951107 = queryWeight, product of:
                3.3540566 = boost
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.021920273 = queryNorm
              0.7271987 = fieldWeight in 3333, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.078125 = fieldNorm(doc=3333)
        0.36 = coord(9/25)
    
  2. Srihari, R.K.: Using speech input for image interpretation, annotation, and retrieval (1997) 0.21
    0.21425734 = sum of:
      0.21425734 = product of:
        0.89273894 = sum of:
          0.138786 = weight(abstract_txt:tell in 1765) [ClassicSimilarity], result of:
            0.138786 = score(doc=1765,freq=1.0), product of:
              0.18167599 = queryWeight, product of:
                1.017126 = boost
                8.148484 = idf(docFreq=33, maxDocs=43254)
                0.021920273 = queryNorm
              0.7639204 = fieldWeight in 1765, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.148484 = idf(docFreq=33, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
          0.037797198 = weight(abstract_txt:language in 1765) [ClassicSimilarity], result of:
            0.037797198 = score(doc=1765,freq=1.0), product of:
              0.09617234 = queryWeight, product of:
                1.0465647 = boost
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.021920273 = queryNorm
              0.39301527 = fieldWeight in 1765, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
          0.06832558 = weight(abstract_txt:natural in 1765) [ClassicSimilarity], result of:
            0.06832558 = score(doc=1765,freq=1.0), product of:
              0.14271381 = queryWeight, product of:
                1.2748938 = boost
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.021920273 = queryNorm
              0.4787594 = fieldWeight in 1765, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
          0.13525617 = weight(abstract_txt:vision in 1765) [ClassicSimilarity], result of:
            0.13525617 = score(doc=1765,freq=1.0), product of:
              0.22499965 = queryWeight, product of:
                1.600781 = boost
                6.4121547 = idf(docFreq=192, maxDocs=43254)
                0.021920273 = queryNorm
              0.6011395 = fieldWeight in 1765, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                6.4121547 = idf(docFreq=192, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
          0.06745357 = weight(abstract_txt:show in 1765) [ClassicSimilarity], result of:
            0.06745357 = score(doc=1765,freq=1.0), product of:
              0.16197357 = queryWeight, product of:
                1.663446 = boost
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.021920273 = queryNorm
              0.416448 = fieldWeight in 1765, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
          0.44512042 = weight(abstract_txt:image in 1765) [ClassicSimilarity], result of:
            0.44512042 = score(doc=1765,freq=5.0), product of:
              0.3951107 = queryWeight, product of:
                3.3540566 = boost
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.021920273 = queryNorm
              1.1265714 = fieldWeight in 1765, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.09375 = fieldNorm(doc=1765)
        0.24 = coord(6/25)
    
  3. Donahue, J.; Hendricks, L.A.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description (2014) 0.21
    0.21311706 = sum of:
      0.21311706 = product of:
        0.6659908 = sum of:
          0.025198134 = weight(abstract_txt:language in 3338) [ClassicSimilarity], result of:
            0.025198134 = score(doc=3338,freq=1.0), product of:
              0.09617234 = queryWeight, product of:
                1.0465647 = boost
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.021920273 = queryNorm
              0.2620102 = fieldWeight in 3338, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.027221046 = weight(abstract_txt:current in 3338) [ClassicSimilarity], result of:
            0.027221046 = score(doc=3338,freq=1.0), product of:
              0.10125298 = queryWeight, product of:
                1.0738531 = boost
                4.3014708 = idf(docFreq=1592, maxDocs=43254)
                0.021920273 = queryNorm
              0.26884192 = fieldWeight in 3338, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.3014708 = idf(docFreq=1592, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.23640203 = weight(abstract_txt:recurrent in 3338) [ClassicSimilarity], result of:
            0.23640203 = score(doc=3338,freq=4.0), product of:
              0.213899 = queryWeight, product of:
                1.1036475 = boost
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.021920273 = queryNorm
              1.105204 = fieldWeight in 3338, product of:
                2.0 = tf(freq=4.0), with freq of:
                  4.0 = termFreq=4.0
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.054939285 = weight(abstract_txt:state in 3338) [ClassicSimilarity], result of:
            0.054939285 = score(doc=3338,freq=2.0), product of:
              0.12834628 = queryWeight, product of:
                1.2090176 = boost
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.021920273 = queryNorm
              0.42805514 = fieldWeight in 3338, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.045550387 = weight(abstract_txt:natural in 3338) [ClassicSimilarity], result of:
            0.045550387 = score(doc=3338,freq=1.0), product of:
              0.14271381 = queryWeight, product of:
                1.2748938 = boost
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.021920273 = queryNorm
              0.31917295 = fieldWeight in 3338, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                5.106767 = idf(docFreq=711, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.044969045 = weight(abstract_txt:show in 3338) [ClassicSimilarity], result of:
            0.044969045 = score(doc=3338,freq=1.0), product of:
              0.16197357 = queryWeight, product of:
                1.663446 = boost
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.021920273 = queryNorm
              0.277632 = fieldWeight in 3338, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.044031672 = weight(abstract_txt:model in 3338) [ClassicSimilarity], result of:
            0.044031672 = score(doc=3338,freq=1.0), product of:
              0.17578888 = queryWeight, product of:
                2.0010211 = boost
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.021920273 = queryNorm
              0.2504804 = fieldWeight in 3338, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
          0.18767925 = weight(abstract_txt:image in 3338) [ClassicSimilarity], result of:
            0.18767925 = score(doc=3338,freq=2.0), product of:
              0.3951107 = queryWeight, product of:
                3.3540566 = boost
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.021920273 = queryNorm
              0.4750042 = fieldWeight in 3338, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.0625 = fieldNorm(doc=3338)
        0.32 = coord(8/25)
    
  4. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; Yuille, A.L.: Explain images with multimodal recurrent neural networks (2014) 0.19
    0.19007033 = sum of:
      0.19007033 = product of:
        0.7919597 = sum of:
          0.013786824 = weight(abstract_txt:from in 3022) [ClassicSimilarity], result of:
            0.013786824 = score(doc=3022,freq=1.0), product of:
              0.06346552 = queryWeight, product of:
                1.0412513 = boost
                2.7805862 = idf(docFreq=7289, maxDocs=43254)
                0.021920273 = queryNorm
              0.2172333 = fieldWeight in 3022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                2.7805862 = idf(docFreq=7289, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
          0.131129 = weight(abstract_txt:generative in 3022) [ClassicSimilarity], result of:
            0.131129 = score(doc=3022,freq=1.0), product of:
              0.19753942 = queryWeight, product of:
                1.060603 = boost
                8.496791 = idf(docFreq=23, maxDocs=43254)
                0.021920273 = queryNorm
              0.6638118 = fieldWeight in 3022, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                8.496791 = idf(docFreq=23, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
          0.20895185 = weight(abstract_txt:recurrent in 3022) [ClassicSimilarity], result of:
            0.20895185 = score(doc=3022,freq=2.0), product of:
              0.213899 = queryWeight, product of:
                1.1036475 = boost
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.021920273 = queryNorm
              0.97687155 = fieldWeight in 3022, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                8.841632 = idf(docFreq=16, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
          0.06867411 = weight(abstract_txt:state in 3022) [ClassicSimilarity], result of:
            0.06867411 = score(doc=3022,freq=2.0), product of:
              0.12834628 = queryWeight, product of:
                1.2090176 = boost
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.021920273 = queryNorm
              0.5350689 = fieldWeight in 3022, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
          0.13481891 = weight(abstract_txt:model in 3022) [ClassicSimilarity], result of:
            0.13481891 = score(doc=3022,freq=6.0), product of:
              0.17578888 = queryWeight, product of:
                2.0010211 = boost
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.021920273 = queryNorm
              0.76693654 = fieldWeight in 3022, product of:
                2.4494898 = tf(freq=6.0), with freq of:
                  6.0 = termFreq=6.0
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
          0.23459905 = weight(abstract_txt:image in 3022) [ClassicSimilarity], result of:
            0.23459905 = score(doc=3022,freq=2.0), product of:
              0.3951107 = queryWeight, product of:
                3.3540566 = boost
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.021920273 = queryNorm
              0.59375525 = fieldWeight in 3022, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.078125 = fieldNorm(doc=3022)
        0.24 = coord(6/25)
    
  5. Ayadi, H.; Torjmen-Khemakhem, M.; Daoud, M.; Xiangji Huang, J.; Ben Jemaa, M.: MF-Re-Rank : a modality feature-based re-ranking model for medical image retrieval (2018) 0.17
    0.16973062 = sum of:
      0.16973062 = product of:
        0.7072109 = sum of:
          0.016715635 = weight(abstract_txt:from in 460) [ClassicSimilarity], result of:
            0.016715635 = score(doc=460,freq=3.0), product of:
              0.06346552 = queryWeight, product of:
                1.0412513 = boost
                2.7805862 = idf(docFreq=7289, maxDocs=43254)
                0.021920273 = queryNorm
              0.26338136 = fieldWeight in 460, product of:
                1.7320508 = tf(freq=3.0), with freq of:
                  3.0 = termFreq=3.0
                2.7805862 = idf(docFreq=7289, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
          0.022048365 = weight(abstract_txt:language in 460) [ClassicSimilarity], result of:
            0.022048365 = score(doc=460,freq=1.0), product of:
              0.09617234 = queryWeight, product of:
                1.0465647 = boost
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.021920273 = queryNorm
              0.22925891 = fieldWeight in 460, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.192163 = idf(docFreq=1776, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
          0.033991948 = weight(abstract_txt:state in 460) [ClassicSimilarity], result of:
            0.033991948 = score(doc=460,freq=1.0), product of:
              0.12834628 = queryWeight, product of:
                1.2090176 = boost
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.021920273 = queryNorm
              0.2648456 = fieldWeight in 460, product of:
                1.0 = tf(freq=1.0), with freq of:
                  1.0 = termFreq=1.0
                4.842891 = idf(docFreq=926, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
          0.055646352 = weight(abstract_txt:show in 460) [ClassicSimilarity], result of:
            0.055646352 = score(doc=460,freq=2.0), product of:
              0.16197357 = queryWeight, product of:
                1.663446 = boost
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.021920273 = queryNorm
              0.34355205 = fieldWeight in 460, product of:
                1.4142135 = tf(freq=2.0), with freq of:
                  2.0 = termFreq=2.0
                4.442112 = idf(docFreq=1383, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
          0.08615059 = weight(abstract_txt:model in 460) [ClassicSimilarity], result of:
            0.08615059 = score(doc=460,freq=5.0), product of:
              0.17578888 = queryWeight, product of:
                2.0010211 = boost
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.021920273 = queryNorm
              0.49007985 = fieldWeight in 460, product of:
                2.236068 = tf(freq=5.0), with freq of:
                  5.0 = termFreq=5.0
                4.0076866 = idf(docFreq=2136, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
          0.492658 = weight(abstract_txt:image in 460) [ClassicSimilarity], result of:
            0.492658 = score(doc=460,freq=18.0), product of:
              0.3951107 = queryWeight, product of:
                3.3540566 = boost
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.021920273 = queryNorm
              1.246886 = fieldWeight in 460, product of:
                4.2426405 = tf(freq=18.0), with freq of:
                  18.0 = termFreq=18.0
                5.374059 = idf(docFreq=544, maxDocs=43254)
                0.0546875 = fieldNorm(doc=460)
        0.24 = coord(6/25)