Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation

最新推荐文章于 2022-06-28 13:39:21 发布

转载最新推荐文章于 2022-06-28 13:39:21 发布 · 893 阅读

NLP 专栏收录该内容

43 篇文章

订阅专栏

本文提出使用多模态紧致双线性池化方法增强神经机器翻译中的注意力机制。该方法通过外积结合视觉与文本特征，在图像字幕翻译任务上取得显著提升。

Multimodal Compact Bilinear Pooling for Multimodal Neural Machine Translation

Jean-Benoit Delbrouck, Stephane Dupont

(Submitted on 23 Mar 2017)

In state-of-the-art Neural Machine Translation, an attention mechanism is used during decoding to enhance the translation. At every step, the decoder uses this mechanism to focus on different parts of the source sentence to gather the most useful information before outputting its target word. Recently, the effectiveness of the attention mechanism has also been explored for multimodal tasks, where it becomes possible to focus both on sentence parts and image regions. Approaches to pool two modalities usually include element-wise product, sum or concatenation. In this paper, we evaluate the more advanced Multimodal Compact Bilinear pooling method, which takes the outer product of two vectors to combine the attention features for the two modalities. This has been previously investigated for visual question answering. We try out this approach for multimodal image caption translation and show improvements compared to basic combination methods.

Comments:	Submitted to ICLR Workshop 2017
Subjects:	Computation and Language (cs.CL)
Cite as:	arXiv:1703.08084 [cs.CL]
	(or arXiv:1703.08084v1 [cs.CL] for this version)