Spatio-Temporal Attention Models for Grounded Video Captioning

最新推荐文章于 2024-10-10 08:24:47 发布

转载最新推荐文章于 2024-10-10 08:24:47 发布 · 1.2k 阅读

paper reading 同时被 2 个专栏收录

85 篇文章

订阅专栏

caption

16 篇文章

订阅专栏

本文提出一种结合时空注意力机制与深度神经网络的视频自动生成字幕模型。该模型能够定位视频中的视觉概念，如主体、动作及对象，并在无需额外监督的情况下，在时间和空间上进行定位。

Spatio-Temporal Attention Models for Grounded Video Captioning

Mihai Zanfir, Elisabeta Marinoiu, Cristian Sminchisescu

(Submitted on 17 Oct 2016 ( v1), last revised 18 Oct 2016 (this version, v2))

Automatic video captioning is challenging due to the complex interactions in dynamic real scenes. A comprehensive system would ultimately localize and track the objects, actions and interactions present in a video and generate a description that relies on temporal localization in order to ground the visual concepts. However, most existing automatic video captioning systems map from raw video data to high level textual description, bypassing localization and recognition, thus discarding potentially valuable information for content localization and generalization. In this work we present an automatic video captioning model that combines spatio-temporal attention and image classification by means of deep neural network structures based on long short-term memory. The resulting system is demonstrated to produce state-of-the-art results in the standard YouTube captioning benchmark while also offering the advantage of localizing the visual concepts (subjects, verbs, objects), with no grounding supervision, over space and time.

Comments:	To appear in Asian Conference on Computer Vision (ACCV), Taipei, Taiwan, 2016
Subjects:	Computer Vision and Pattern Recognition (cs.CV)
Cite as:	arXiv:1610.04997 [cs.CV]
	(or arXiv:1610.04997v2 [cs.CV] for this version)