Document (#38875)

Author

Vinyals, O.
Toshev, A.
Bengio, S.
Erhan, D.

Title

¬A picture is worth a thousand (coherent) words : building a natural description of images

Source

http://googleresearch.blogspot.de/2014/11/a-picture-is-worth-thousand-coherent.html

Year

2014

Content

"People can summarize a complex scene in a few words without thinking twice. It's much more difficult for computers. But we've just gotten a bit closer -- we've developed a machine-learning system that can automatically produce captions (like the three above) to accurately describe images the first time it sees them. This kind of system could eventually help visually impaired people understand pictures, provide alternate text for images in parts of the world where mobile connections are slow, and make it easier for everyone to search on Google for images. Recent research has greatly improved object detection, classification, and labeling. But accurately describing a complex scene requires a deeper representation of what's going on in the scene, capturing how the various objects relate to one another and translating it all into natural-sounding language. Many efforts to construct computer-generated natural descriptions of images propose combining current state-of-the-art techniques in both computer vision and natural language processing to form a complete image description approach. But what if we instead merged recent computer vision and language models into a single jointly trained system, taking an image and directly producing a human readable sequence of words to describe it? This idea comes from recent advances in machine translation between languages, where a Recurrent Neural Network (RNN) transforms, say, a French sentence into a vector representation, and a second RNN uses that vector representation to generate a target sentence in German. Now, what if we replaced that first RNN and its input words with a deep Convolutional Neural Network (CNN) trained to classify objects in images? Normally, the CNN's last layer is used in a final Softmax among known classes of objects, assigning a probability that each object might be in the image. But if we remove that final layer, we can instead feed the CNN's rich encoding of the image into a RNN designed to produce phrases. We can then train the whole system directly on images and their captions, so it maximizes the likelihood that descriptions it produces best match the training descriptions for each image.
Our experiments with this system on several openly published datasets, including Pascal, Flickr8k, Flickr30k and SBU, show how robust the qualitative results are -- the generated sentences are quite reasonable. It also performs well in quantitative evaluations with the Bilingual Evaluation Understudy (BLEU), a metric used in machine translation to evaluate the quality of generated sentences. A picture may be worth a thousand words, but sometimes it's the words that are most useful -- so it's important we figure out ways to translate from images to words automatically and accurately. As the datasets suited to learning image descriptions grow and mature, so will the performance of end-to-end approaches like this. We look forward to continuing developments in systems that can read images and generate good natural-language descriptions. To get more details about the framework used to generate descriptions from images, as well as the model evaluation, read the full paper here." Vgl. auch: https://news.ycombinator.com/item?id=8621658.

Footnote

Vgl.: http://arxiv.org/abs/1411.4555.

Theme

Automatisches Indexieren

Form

Bilder

Object

Google

Similar documents (content)

1. Graphic details : a scientific study of the importance of diagrams to science (2016) 0.54

   0.5405302 = sum of:
     0.5405302 = product of:
       1.216193 = sum of:
         0.13314636 = weight(abstract_txt:words in 3035) [ClassicSimilarity], result of:
           0.13314636 = score(doc=3035,freq=1.0), product of:
             0.22741205 = queryWeight, product of:
               1.1070966 = boost
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.038373392 = queryNorm
             0.5854851 = fieldWeight in 3035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.109375 = fieldNorm(doc=3035)
         0.22686566 = weight(abstract_txt:picture in 3035) [ClassicSimilarity], result of:
           0.22686566 = score(doc=3035,freq=1.0), product of:
             0.3244191 = queryWeight, product of:
               1.322306 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.038373392 = queryNorm
             0.6992981 = fieldWeight in 3035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.109375 = fieldNorm(doc=3035)
         0.34150153 = weight(abstract_txt:worth in 3035) [ClassicSimilarity], result of:
           0.34150153 = score(doc=3035,freq=1.0), product of:
             0.42611063 = queryWeight, product of:
               1.5154442 = boost
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.038373392 = queryNorm
             0.8014387 = fieldWeight in 3035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.109375 = fieldNorm(doc=3035)
         0.5146794 = weight(abstract_txt:thousand in 3035) [ClassicSimilarity], result of:
           0.5146794 = score(doc=3035,freq=1.0), product of:
             0.5601252 = queryWeight, product of:
               1.737486 = boost
               8.401051 = idf(docFreq=26, maxDocs=44218)
               0.038373392 = queryNorm
             0.9188649 = fieldWeight in 3035, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.401051 = idf(docFreq=26, maxDocs=44218)
               0.109375 = fieldNorm(doc=3035)
       0.44444445 = coord(4/9)
   

2. Rolling, L.: ¬The role of graphic display of concept relationships in indexing and retrieval vocabularies (1985) 0.23

   0.2316558 = sum of:
     0.2316558 = product of:
       0.5212256 = sum of:
         0.05706273 = weight(abstract_txt:words in 3646) [ClassicSimilarity], result of:
           0.05706273 = score(doc=3646,freq=1.0), product of:
             0.22741205 = queryWeight, product of:
               1.1070966 = boost
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.038373392 = queryNorm
             0.2509222 = fieldWeight in 3646, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.353007 = idf(docFreq=568, maxDocs=44218)
               0.046875 = fieldNorm(doc=3646)
         0.09722815 = weight(abstract_txt:picture in 3646) [ClassicSimilarity], result of:
           0.09722815 = score(doc=3646,freq=1.0), product of:
             0.3244191 = queryWeight, product of:
               1.322306 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.038373392 = queryNorm
             0.2996992 = fieldWeight in 3646, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.046875 = fieldNorm(doc=3646)
         0.1463578 = weight(abstract_txt:worth in 3646) [ClassicSimilarity], result of:
           0.1463578 = score(doc=3646,freq=1.0), product of:
             0.42611063 = queryWeight, product of:
               1.5154442 = boost
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.038373392 = queryNorm
             0.34347373 = fieldWeight in 3646, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.046875 = fieldNorm(doc=3646)
         0.22057687 = weight(abstract_txt:thousand in 3646) [ClassicSimilarity], result of:
           0.22057687 = score(doc=3646,freq=1.0), product of:
             0.5601252 = queryWeight, product of:
               1.737486 = boost
               8.401051 = idf(docFreq=26, maxDocs=44218)
               0.038373392 = queryNorm
             0.39379925 = fieldWeight in 3646, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.401051 = idf(docFreq=26, maxDocs=44218)
               0.046875 = fieldNorm(doc=3646)
       0.44444445 = coord(4/9)
   

3. Jascó, P.: Searching for images on the Web : pt.2 (1997) 0.23

   0.22830102 = sum of:
     0.22830102 = product of:
       0.684903 = sum of:
         0.16251628 = weight(abstract_txt:natural in 548) [ClassicSimilarity], result of:
           0.16251628 = score(doc=548,freq=1.0), product of:
             0.20476569 = queryWeight, product of:
               1.0505273 = boost
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.038373392 = queryNorm
             0.79366946 = fieldWeight in 548, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0794845 = idf(docFreq=747, maxDocs=44218)
               0.15625 = fieldNorm(doc=548)
         0.19829297 = weight(abstract_txt:images in 548) [ClassicSimilarity], result of:
           0.19829297 = score(doc=548,freq=1.0), product of:
             0.23381057 = queryWeight, product of:
               1.1225632 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.038373392 = queryNorm
             0.8480924 = fieldWeight in 548, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.15625 = fieldNorm(doc=548)
         0.3240938 = weight(abstract_txt:picture in 548) [ClassicSimilarity], result of:
           0.3240938 = score(doc=548,freq=1.0), product of:
             0.3244191 = queryWeight, product of:
               1.322306 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.038373392 = queryNorm
             0.9989972 = fieldWeight in 548, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.15625 = fieldNorm(doc=548)
       0.33333334 = coord(3/9)
   

4. Cawkell, A.E.: Developments in indexing picture collections (1993) 0.18

   0.18297394 = sum of:
     0.18297394 = product of:
       0.82338274 = sum of:
         0.23795155 = weight(abstract_txt:images in 7445) [ClassicSimilarity], result of:
           0.23795155 = score(doc=7445,freq=1.0), product of:
             0.23381057 = queryWeight, product of:
               1.1225632 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.038373392 = queryNorm
             1.0177108 = fieldWeight in 7445, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.1875 = fieldNorm(doc=7445)
         0.5854312 = weight(abstract_txt:worth in 7445) [ClassicSimilarity], result of:
           0.5854312 = score(doc=7445,freq=1.0), product of:
             0.42611063 = queryWeight, product of:
               1.5154442 = boost
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.038373392 = queryNorm
             1.3738949 = fieldWeight in 7445, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.3274393 = idf(docFreq=78, maxDocs=44218)
               0.1875 = fieldNorm(doc=7445)
       0.22222222 = coord(2/9)
   

5. Cawkell, A.E.: Imaging systems and picture collection management : a review (1992) 0.17

   0.17376304 = sum of:
     0.17376304 = product of:
       0.5212891 = sum of:
         0.09812346 = weight(abstract_txt:description in 5097) [ClassicSimilarity], result of:
           0.09812346 = score(doc=5097,freq=1.0), product of:
             0.1855421 = queryWeight, product of:
               4.835176 = idf(docFreq=954, maxDocs=44218)
               0.038373392 = queryNorm
             0.5288474 = fieldWeight in 5097, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.835176 = idf(docFreq=954, maxDocs=44218)
               0.109375 = fieldNorm(doc=5097)
         0.19630001 = weight(abstract_txt:images in 5097) [ClassicSimilarity], result of:
           0.19630001 = score(doc=5097,freq=2.0), product of:
             0.23381057 = queryWeight, product of:
               1.1225632 = boost
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.038373392 = queryNorm
             0.8395686 = fieldWeight in 5097, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.427791 = idf(docFreq=527, maxDocs=44218)
               0.109375 = fieldNorm(doc=5097)
         0.22686566 = weight(abstract_txt:picture in 5097) [ClassicSimilarity], result of:
           0.22686566 = score(doc=5097,freq=1.0), product of:
             0.3244191 = queryWeight, product of:
               1.322306 = boost
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.038373392 = queryNorm
             0.6992981 = fieldWeight in 5097, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.3935823 = idf(docFreq=200, maxDocs=44218)
               0.109375 = fieldNorm(doc=5097)
       0.33333334 = coord(3/9)