Karpathy, A.; Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions (2015)
0.01
0.0063645868 = product of:
0.01909376 = sum of:
0.01909376 = product of:
0.057281278 = sum of:
0.057281278 = weight(_text_:network in 1868) [ClassicSimilarity], result of:
0.057281278 = score(doc=1868,freq=2.0), product of:
0.19402927 = queryWeight, product of:
4.4533744 = idf(docFreq=1398, maxDocs=44218)
0.043569047 = queryNorm
0.29521978 = fieldWeight in 1868, product of:
1.4142135 = tf(freq=2.0), with freq of:
2.0 = termFreq=2.0
4.4533744 = idf(docFreq=1398, maxDocs=44218)
0.046875 = fieldNorm(doc=1868)
0.33333334 = coord(1/3)
0.33333334 = coord(1/3)
- Abstract
- We present a model that generates free-form natural language descriptions of image regions. Our model leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between text and visual data. Our approach is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate the effectiveness of our alignment model with ranking experiments on Flickr8K, Flickr30K and COCO datasets, where we substantially improve on the state of the art. We then show that the sentences created by our generative model outperform retrieval baselines on the three aforementioned datasets and a new dataset of region-level annotations.
