Document (#38870)

Author

Vinyals, O.
Toshev, A.
Bengio, S.
Erhan, D.

Title

Show and tell : a neural image caption generator

Source

http://arxiv.org/abs/1411.4555

Year

2014

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

Content

Vgl.: http://googleresearch.blogspot.de/2014/11/a-picture-is-worth-thousand-coherent.html. Vgl. auch: https://news.ycombinator.com/item?id=8621658.

Theme

Automatisches Indexieren

Form

Bilder

Object

Google

Similar documents (content)

1. Karpathy, A.; Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions (2015) 0.38

   0.37747702 = sum of:
     0.37747702 = product of:
       1.0485473 = sum of:
         0.12073248 = weight(abstract_txt:generative in 1868) [ClassicSimilarity], result of:
           0.12073248 = score(doc=1868,freq=1.0), product of:
             0.18774721 = queryWeight, product of:
               1.0372324 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.021990588 = queryNorm
             0.6430587 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.03167026 = weight(abstract_txt:language in 1868) [ClassicSimilarity], result of:
           0.03167026 = score(doc=1868,freq=1.0), product of:
             0.0969323 = queryWeight, product of:
               1.053995 = boost
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.021990588 = queryNorm
             0.32672557 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.20169169 = weight(abstract_txt:recurrent in 1868) [ClassicSimilarity], result of:
           0.20169169 = score(doc=1868,freq=2.0), product of:
             0.20980029 = queryWeight, product of:
               1.0964589 = boost
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.021990588 = queryNorm
             0.96135086 = fieldWeight in 1868, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.048786692 = weight(abstract_txt:state in 1868) [ClassicSimilarity], result of:
           0.048786692 = score(doc=1868,freq=1.0), product of:
             0.12929101 = queryWeight, product of:
               1.2172743 = boost
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.021990588 = queryNorm
             0.37734017 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.056745477 = weight(abstract_txt:natural in 1868) [ClassicSimilarity], result of:
           0.056745477 = score(doc=1868,freq=1.0), product of:
             0.14299524 = queryWeight, product of:
               1.2801625 = boost
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.021990588 = queryNorm
             0.39683473 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.05554835 = weight(abstract_txt:show in 1868) [ClassicSimilarity], result of:
           0.05554835 = score(doc=1868,freq=1.0), product of:
             0.16137835 = queryWeight, product of:
               1.6656072 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.021990588 = queryNorm
             0.3442119 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.13263464 = weight(abstract_txt:dataset in 1868) [ClassicSimilarity], result of:
           0.13263464 = score(doc=1868,freq=1.0), product of:
             0.25184813 = queryWeight, product of:
               1.6989229 = boost
               6.7410603 = idf(docFreq=141, maxDocs=44218)
               0.021990588 = queryNorm
             0.5266453 = fieldWeight in 1868, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.7410603 = idf(docFreq=141, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.10970355 = weight(abstract_txt:model in 1868) [ClassicSimilarity], result of:
           0.10970355 = score(doc=1868,freq=4.0), product of:
             0.17613184 = queryWeight, product of:
               2.0092697 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.021990588 = queryNorm
             0.62284905 = fieldWeight in 1868, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
         0.29103416 = weight(abstract_txt:image in 1868) [ClassicSimilarity], result of:
           0.29103416 = score(doc=1868,freq=3.0), product of:
             0.40019315 = queryWeight, product of:
               3.386173 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.021990588 = queryNorm
             0.72723424 = fieldWeight in 1868, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.078125 = fieldNorm(doc=1868)
       0.36 = coord(9/25)
   

2. Srihari, R.K.: Using speech input for image interpretation, annotation, and retrieval (1997) 0.22

   0.21590863 = sum of:
     0.21590863 = product of:
       0.89961934 = sum of:
         0.14019844 = weight(abstract_txt:tell in 764) [ClassicSimilarity], result of:
           0.14019844 = score(doc=764,freq=1.0), product of:
             0.18368149 = queryWeight, product of:
               1.0259401 = boost
               8.14154 = idf(docFreq=34, maxDocs=44218)
               0.021990588 = queryNorm
             0.7632693 = fieldWeight in 764, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.14154 = idf(docFreq=34, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
         0.038004316 = weight(abstract_txt:language in 764) [ClassicSimilarity], result of:
           0.038004316 = score(doc=764,freq=1.0), product of:
             0.0969323 = queryWeight, product of:
               1.053995 = boost
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.021990588 = queryNorm
             0.3920707 = fieldWeight in 764, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
         0.06809457 = weight(abstract_txt:natural in 764) [ClassicSimilarity], result of:
           0.06809457 = score(doc=764,freq=1.0), product of:
             0.14299524 = queryWeight, product of:
               1.2801625 = boost
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.021990588 = queryNorm
             0.47620165 = fieldWeight in 764, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
         0.13579583 = weight(abstract_txt:vision in 764) [ClassicSimilarity], result of:
           0.13579583 = score(doc=764,freq=1.0), product of:
             0.22655354 = queryWeight, product of:
               1.6113495 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.021990588 = queryNorm
             0.5993984 = fieldWeight in 764, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
         0.06665802 = weight(abstract_txt:show in 764) [ClassicSimilarity], result of:
           0.06665802 = score(doc=764,freq=1.0), product of:
             0.16137835 = queryWeight, product of:
               1.6656072 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.021990588 = queryNorm
             0.4130543 = fieldWeight in 764, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
         0.45086822 = weight(abstract_txt:image in 764) [ClassicSimilarity], result of:
           0.45086822 = score(doc=764,freq=5.0), product of:
             0.40019315 = queryWeight, product of:
               3.386173 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.021990588 = queryNorm
             1.1266265 = fieldWeight in 764, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.09375 = fieldNorm(doc=764)
       0.24 = coord(6/25)
   

3. Donahue, J.; Hendricks, L.A.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description (2014) 0.21

   0.2112138 = sum of:
     0.2112138 = product of:
       0.6600431 = sum of:
         0.02533621 = weight(abstract_txt:language in 1873) [ClassicSimilarity], result of:
           0.02533621 = score(doc=1873,freq=1.0), product of:
             0.0969323 = queryWeight, product of:
               1.053995 = boost
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.021990588 = queryNorm
             0.26138046 = fieldWeight in 1873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.027503764 = weight(abstract_txt:current in 1873) [ClassicSimilarity], result of:
           0.027503764 = score(doc=1873,freq=1.0), product of:
             0.102384806 = queryWeight, product of:
               1.0832335 = boost
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.021990588 = queryNorm
             0.2686313 = fieldWeight in 1873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.298101 = idf(docFreq=1633, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.2281881 = weight(abstract_txt:recurrent in 1873) [ClassicSimilarity], result of:
           0.2281881 = score(doc=1873,freq=4.0), product of:
             0.20980029 = queryWeight, product of:
               1.0964589 = boost
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.021990588 = queryNorm
             1.0876443 = fieldWeight in 1873, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.05519584 = weight(abstract_txt:state in 1873) [ClassicSimilarity], result of:
           0.05519584 = score(doc=1873,freq=2.0), product of:
             0.12929101 = queryWeight, product of:
               1.2172743 = boost
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.021990588 = queryNorm
             0.42691165 = fieldWeight in 1873, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.04539638 = weight(abstract_txt:natural in 1873) [ClassicSimilarity], result of:
           0.04539638 = score(doc=1873,freq=1.0), product of:
             0.14299524 = queryWeight, product of:
               1.2801625 = boost
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.021990588 = queryNorm
             0.31746778 = fieldWeight in 1873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.04443868 = weight(abstract_txt:show in 1873) [ClassicSimilarity], result of:
           0.04443868 = score(doc=1873,freq=1.0), product of:
             0.16137835 = queryWeight, product of:
               1.6656072 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.021990588 = queryNorm
             0.27536952 = fieldWeight in 1873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.04388142 = weight(abstract_txt:model in 1873) [ClassicSimilarity], result of:
           0.04388142 = score(doc=1873,freq=1.0), product of:
             0.17613184 = queryWeight, product of:
               2.0092697 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.021990588 = queryNorm
             0.24913962 = fieldWeight in 1873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
         0.19010273 = weight(abstract_txt:image in 1873) [ClassicSimilarity], result of:
           0.19010273 = score(doc=1873,freq=2.0), product of:
             0.40019315 = queryWeight, product of:
               3.386173 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.021990588 = queryNorm
             0.47502744 = fieldWeight in 1873, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.0625 = fieldNorm(doc=1873)
       0.32 = coord(8/25)
   

4. Mao, J.; Xu, W.; Yang, Y.; Wang, J.; Yuille, A.L.: Explain images with multimodal recurrent neural networks (2014) 0.19

   0.18650854 = sum of:
     0.18650854 = product of:
       0.7771189 = sum of:
         0.12073248 = weight(abstract_txt:generative in 1557) [ClassicSimilarity], result of:
           0.12073248 = score(doc=1557,freq=1.0), product of:
             0.18774721 = queryWeight, product of:
               1.0372324 = boost
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.021990588 = queryNorm
             0.6430587 = fieldWeight in 1557, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.231152 = idf(docFreq=31, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
         0.013712682 = weight(abstract_txt:from in 1557) [ClassicSimilarity], result of:
           0.013712682 = score(doc=1557,freq=1.0), product of:
             0.06350571 = queryWeight, product of:
               1.0448557 = boost
               2.7638826 = idf(docFreq=7577, maxDocs=44218)
               0.021990588 = queryNorm
             0.21592833 = fieldWeight in 1557, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.7638826 = idf(docFreq=7577, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
         0.20169169 = weight(abstract_txt:recurrent in 1557) [ClassicSimilarity], result of:
           0.20169169 = score(doc=1557,freq=2.0), product of:
             0.20980029 = queryWeight, product of:
               1.0964589 = boost
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.021990588 = queryNorm
             0.96135086 = fieldWeight in 1557, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.701155 = idf(docFreq=19, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
         0.0689948 = weight(abstract_txt:state in 1557) [ClassicSimilarity], result of:
           0.0689948 = score(doc=1557,freq=2.0), product of:
             0.12929101 = queryWeight, product of:
               1.2172743 = boost
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.021990588 = queryNorm
             0.53363955 = fieldWeight in 1557, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
         0.13435887 = weight(abstract_txt:model in 1557) [ClassicSimilarity], result of:
           0.13435887 = score(doc=1557,freq=6.0), product of:
             0.17613184 = queryWeight, product of:
               2.0092697 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.021990588 = queryNorm
             0.7628312 = fieldWeight in 1557, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
         0.23762843 = weight(abstract_txt:image in 1557) [ClassicSimilarity], result of:
           0.23762843 = score(doc=1557,freq=2.0), product of:
             0.40019315 = queryWeight, product of:
               3.386173 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.021990588 = queryNorm
             0.59378433 = fieldWeight in 1557, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.078125 = fieldNorm(doc=1557)
       0.24 = coord(6/25)
   

5. Ayadi, H.; Torjmen-Khemakhem, M.; Daoud, M.; Xiangji Huang, J.; Ben Jemaa, M.: MF-Re-Rank : a modality feature-based re-ranking model for medical image retrieval (2018) 0.17

   0.17107487 = sum of:
     0.17107487 = product of:
       0.71281195 = sum of:
         0.016625743 = weight(abstract_txt:from in 4459) [ClassicSimilarity], result of:
           0.016625743 = score(doc=4459,freq=3.0), product of:
             0.06350571 = queryWeight, product of:
               1.0448557 = boost
               2.7638826 = idf(docFreq=7577, maxDocs=44218)
               0.021990588 = queryNorm
             0.2617992 = fieldWeight in 4459, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.7638826 = idf(docFreq=7577, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
         0.022169184 = weight(abstract_txt:language in 4459) [ClassicSimilarity], result of:
           0.022169184 = score(doc=4459,freq=1.0), product of:
             0.0969323 = queryWeight, product of:
               1.053995 = boost
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.021990588 = queryNorm
             0.22870791 = fieldWeight in 4459, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.1820874 = idf(docFreq=1834, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
         0.034150682 = weight(abstract_txt:state in 4459) [ClassicSimilarity], result of:
           0.034150682 = score(doc=4459,freq=1.0), product of:
             0.12929101 = queryWeight, product of:
               1.2172743 = boost
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.021990588 = queryNorm
             0.2641381 = fieldWeight in 4459, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.829954 = idf(docFreq=959, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
         0.05499006 = weight(abstract_txt:show in 4459) [ClassicSimilarity], result of:
           0.05499006 = score(doc=4459,freq=2.0), product of:
             0.16137835 = queryWeight, product of:
               1.6656072 = boost
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.021990588 = queryNorm
             0.3407524 = fieldWeight in 4459, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4059124 = idf(docFreq=1466, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
         0.08585661 = weight(abstract_txt:model in 4459) [ClassicSimilarity], result of:
           0.08585661 = score(doc=4459,freq=5.0), product of:
             0.17613184 = queryWeight, product of:
               2.0092697 = boost
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.021990588 = queryNorm
             0.4874565 = fieldWeight in 4459, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               3.986234 = idf(docFreq=2231, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
         0.49901962 = weight(abstract_txt:image in 4459) [ClassicSimilarity], result of:
           0.49901962 = score(doc=4459,freq=18.0), product of:
             0.40019315 = queryWeight, product of:
               3.386173 = boost
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.021990588 = queryNorm
             1.2469469 = fieldWeight in 4459, product of:
               4.2426405 = tf(freq=18.0), with freq of:
                 18.0 = termFreq=18.0
               5.374322 = idf(docFreq=556, maxDocs=44218)
               0.0546875 = fieldNorm(doc=4459)
       0.24 = coord(6/25)