Multimodal Agent with Long-Term Memory: M3-Agent 简介

最新推荐文章于 2025-11-18 02:07:35 发布

原创最新推荐文章于 2025-11-18 02:07:35 发布 · 1k 阅读

14 ·

CC 4.0 BY-SA版权

文章标签：

#ai

大数据专栏收录该内容

36 篇文章

订阅专栏

M3-Agent: 具备长期记忆的多模态智能体
在这里插入图片描述

Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

We introduce M3-Agent, a novel multimodal agent framework equipped with long-term memory. Like humans, M3-Agent can process real-time visual and auditory inputs to build and update its long-term memory. Beyond episodic memory, it also develops semantic memory, enabling it to accumulate world knowledge over time. Its memory is organized in an entity-centric, multimodal format, allowing deeper and more consistent understanding of the environment. Given an instruction, M3-Agent autonomously performs multi-turn, iterative reasoning and retrieves relevant information from memory to accomplish the task.
本文介绍了一种具有（equipped with）长时记忆功能的新型（novel）多模式agent框架M3-Agent。像人类一样，M3-Agent可以处理实时的视觉（visual）和听觉（auditory）输入，以建立和更新其长期记忆。除了情景记忆（episodic memory），它还发展语义记忆（semantic memory），使其能够随着时间的推移积累世界知识。它的记忆以实体为中心（entity-centric）、多模式（multimodal format）的形式组织，允许对环境进行更深入、更一致的理解。在给定指令的情况下（given an instruction），M3-Agent可以自主地（autonomously）进行多轮迭代推理（multi-turn, iterative reasoning），并从记忆中检索相关信息来完成（accomplish）任务。

To evaluate memory effectiveness and memory-based reasoning in multimodal agents, we develop M3-Bench, a new long-video question answering benchmark. M3-Bench comprises 100 newly recorded real-world videos captured from a robot’s perspective (M3-Bench-robot) and 920 web-sourced videos across diverse scenarios (M3-Bench-web). We annotate question-answer pairs designed to test key capabilities essential for agent applications, such as human understanding, general knowledge extraction, and cross- modal reasoning.
为了评估多模态智能体的记忆有效性（effectiveness）和基于记忆的推理，我们开发了一个新的长视频问答基准（question answering benchmark）——M3-Bench。M3-Bench包括（comprise）100个新录制的真实世界视频，从机器人的角度（perspective）（M3-Bench-robot）和920个网络来源的视频跨越不同的场景（diverse scenarios，M3-Bench-web）。我们注释了问答对，旨在（designed to）测试智能体应用程序（essential for agent applications）的关键能力，如人类理解、一般知识提取（extraction）和跨模态推理（cross- modal reasoning）。

Experimental results show that M3-Agent, trained via reinforcement learning, outperforms the strongest baseline, a prompting agent using Gemini-1.5-pro and GPT-4o, achieving 8.2%, 7.7%, and 5.3% higher accuracy on M3-Bench-robot, M3-Bench-web and VideoMME-long, respectively. Our work advances the multimodal agents toward more human-like long-term memory and provides insights into their practical design.
实验结果表明，通过强化（reinforcement）学习训练的M3-Agent优于（outperforms）最强基线，即使用Gemini-1.5-pro和gpt - 40的提示代理，在M3-Bench-robot、M3-Bench-web和VideoMME-long上的准确率分别提高了8.2%、7.7%和5.3%。我们的工作推动了多模式代理向更像人类的长期记忆方向发展，并为它们的实际设计提供了见解。

在这里插入图片描述
示例

https://www.youtube.com/watch?v=XUx31cBanfo