Weakly Supervised Dense Video Captioning

最新推荐文章于 2025-01-01 14:57:11 发布

原创最新推荐文章于 2025-01-01 14:57:11 发布 · 8.4k 阅读

0 ·

CC 4.0 BY-SA版权

paper reading 专栏收录该内容

85 篇文章

订阅专栏

本文提出一种新的密集视频标注方法，该方法可在未明确标注细粒度句子与视频区域对应关系的情况下进行训练，仅依赖于弱化的视频级句子注释。通过引入词汇完全卷积神经网络、子模块最大化方案及序列到序列的学习型语言模型等创新技术，实现了自动为视频片段生成多个信息丰富且多样化的描述。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

https://arxiv.org/abs/1704.01502

Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, Xiangyang Xue

(Submitted on 5 Apr 2017)

This paper focuses on a novel and challenging vision task, dense video captioning, which aims to automatically describe a video clip with multiple informative and diverse caption sentences. The proposed method is trained without explicit annotation of fine-grained sentence to video region-sequence correspondence, but is only based on weak video-level sentence annotations. It differs from existing video captioning systems in three technical aspects. First, we propose lexical fully convolutional neural networks (Lexical-FCN) with weakly supervised multi-instance multi-label learning to weakly link video regions with lexical labels. Second, we introduce a novel submodular maximization scheme to generate multiple informative and diverse region-sequences based on the Lexical-FCN outputs. A winner-takes-all scheme is adopted to weakly associate sentences to region-sequences in the training phase. Third, a sequence-to-sequence learning based language model is trained with the weakly supervised information obtained through the association process. We show that the proposed method can not only produce informative and diverse dense captions, but also outperform state-of-the-art single video captioning methods by a large margin.

Comments:	To appear in CVPR 2017
Subjects:	Computer Vision and Pattern Recognition (cs.CV)
Cite as:	arXiv:1704.01502 [cs.CV]
	(or arXiv:1704.01502v1 [cs.CV] for this version)