An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model

Zhang H, Cao L, Ma J. Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model[J]. arXiv preprint arXiv:2410.23905, 2024.

Summary

• 主要面对的问题：
• composite degradation
• foreground objects--may compromise the delineation of crucial objects
• 提出的算法：
• text-modulated diffusion model (Text-DiFuse)

Method

• a fusion control module (FCM)
• 作用：fusing multiple diffusion processes at the feature level
• a text-controlled fusion re-modulation strategy
• text and a zero-shot location model (OWL-VIT and SAM)

Experiments