Improved Image Captioning via Policy Gradient optimization of SPIDEr

最新推荐文章于 2022-11-18 21:11:43 发布

转载最新推荐文章于 2022-11-18 21:11:43 发布 · 6.9k 阅读

paper reading 专栏收录该内容

85 篇文章

订阅专栏

本文提出一种使用策略梯度方法直接优化SPICE和CIDEr（统称为SPIDEr）组合得分的图像描述生成方法。该方法确保生成的描述在语义上忠于图像且语法流畅。

https://arxiv.org/abs/1704.05539

Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, Kevin Murphy

(Submitted on 1 Dec 2016 ( v1), last revised 18 Mar 2017 (this version, v3))

Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

Comments:	Under review at ICCV 2017
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as:	arXiv:1612.00370 [cs.CV]
	(or arXiv:1612.00370v3 [cs.CV] for this version)