2025_NIPS_ChatVLA-2: Vision-Language-Action Model with Open-World Reasoning

一、文章主要内容总结

本文聚焦机器人领域的视觉-语言-动作(VLA)模型,针对现有VLA系统在微调过程中易丢失预训练视觉-语言模型(VLM)核心能力(如开放世界推理、数学推理、空间推理等)的问题,提出了ChatVLA-2模型。该模型通过动态混合专家(Dynamic MoE)架构和两阶段训练策略,在保留VLM预训练知识的同时,实现了推理与机器人动作的有效对齐。

为验证模型性能,作者设计了两项核心任务:

  1. 数学匹配游戏:机器人识别白板上的数学方程,从候选卡片中选择正确答案并放置,评估模型的OCR、数学推理和操作能力;
  2. 玩具放置任务:机器人根据空间指令(如“将玩具放在目标物体前方/上方”),对未见过的物体执行放置操作,评估空间推理和开放世界物体识别能力。

实验结果表明,ChatVLA-2在开放世界场景中显著优于OpenVLA、DexVLA等现有方法,数学匹配任务操作成功率达82.7%,玩具放置任务达81.4%,同时在12个多模态理解基准中的11个实现性能提升。

二、文章创新点

  1. 动态混合专家(Dynamic MoE)架构:在VLM骨干网络中引入动态MoE,解耦多模态理解与机器人控制的特征空间,既保留共享特征(如空间推理),又让专家模块专注于特定任务,避免预训练知识丢失;
  2. 推理跟随增强模块:将推理令牌通过MLP投影后注入动作专家的后半层,确保机器人动作严格遵循模型内部推理过程,提升对未见过场景的适配性;
船载 OpenHarmony AIS 航海工作台:告警与防碰撞预警系统 船载OpenHarmony AIS航海工作台的告警与防碰撞系统,集成十余类告警(超速、偏航、会遇、SOS等),通过网格空间索引实现高效邻近判定,保障实时性。告警管理器统一调度规则、去重、通知与日志,支持悬浮球、语音、跑马灯、弹窗多渠道触达,确保航行安全。SOS紧急通道独立运行,避免干扰。系统设计强调性能优化与多场景适配,为航海安全提供核心支撑。 阅读详情

相关推荐

AI 应用开发:四维架构落地实战——以 HR Agent 为例拆解感知·推理·记忆·执行【意图路由、规划编排、长期记忆、工具执行、面试题】

本文基于四维架构(感知·推理·记忆·执行),以HR Agent为例,系统拆解大模型应用从“聊天”到“生产可用”的落地路径。通过Spring AI实现意图路由、规划编排、RAG检索与工具调用,结合真实踩坑复盘,揭示架构化设计对安全、成本与可维护性的关键价值,为AI Agent工程化提供可复用的实战框架。

王大师企业官方博客 1325

基于价值的深度强化学习:DQN

本文主要是介绍DQN原理和算法伪代码和代码实现,算法实现包括pytorch版本和stable-baselines3版本

溯源的专栏 1086

Docker小白也能搞定!5分钟用calibre-web搭建私人Kindle书库(附静读天下配置)

本文为零基础用户提供了一份详尽的指南,介绍如何利用Docker容器技术,在5分钟内快速部署Calibre-Web服务,从而搭建一个完全私有的个人Kindle书库。教程涵盖了从Docker安装、Calibre-Web一键部署、书籍元数据管理到移动端配置的全流程,并重点讲解了如何与安卓端知名阅读应用“静读天下”联动,实现跨设备无缝访问,打造媲美在线书城的个性化阅读体验。

weixin_29216957的博客 544

知识图谱的数据挖掘与分析

1. 背景介绍 1.1 数据挖掘的崛起 随着互联网的普及和大数据技术的发展,数据挖掘已经成为了计算机科学领域的热门研究方向。数据挖掘是从大量的、不完全的、有噪声的、模糊的、随机的实际应用数据中,通过计算机科学、统计学、信息论等多学科交叉的方法,挖掘出有用信息和知识的过程。数据挖掘的目标是从数据中发现有

AI天才研究院 1601

AI工程化之生成式UI & A2UI(一)

本文系统介绍了生成式UI和Google A2UI技术。生成式UI通过AI直接生成前端代码(完全生成式)或DSL(混合式),实现千人千面的动态界面和研发提效。Google A2UI提出组件级和消息级协议,采用JSON规范实现渲染框架解耦,支持多端渲染。文章详细阐述了A2UI的架构设计,包括协议生成、服务端处理和渲染器实现,并对比了与传统前端框架的差异,指出A2UI本质是UI=F(AI(UI)+AI(state))的创新模式。最后通过playground项目展示了A2UI的实践应用。

qq_34306228的博客 593

告别网络卡顿!保姆级教程:用国内镜像源5分钟搞定ESP32开发环境(Arduino IDE)

本文提供了一份保姆级教程,教你如何利用国内镜像源在5分钟内快速搭建ESP32开发环境(Arduino IDE),告别网络卡顿问题。教程详细介绍了镜像源配置、开发板安装、测试程序烧录等步骤,并包含与巴法云对接的实战案例,帮助开发者高效完成物联网项目。

weixin_42536058的博客 594

手势识别器:实现手部动作识别器

在本篇文章中,我们将使用Python来实现一个简单的手部动作识别器。然后,我们通过循环从摄像头获取每一帧图像,并对每一帧图像进行处理。我们提取手部关键点并计算手势的特征向量,然后将其添加到特征列表和标签列表中。接下来,我们再次从摄像头获取每一帧图像,并进行手势识别。我们提取手部关键点并计算手势的特征向量,然后使用训练好的分类器来预测手势,并在图像上显示出来。在这个实现中,我们将使用MediaPipe库来检测和跟踪手部关键点,然后使用K最近邻(K-Nearest Neighbors)算法对手势进行分类。

BitSlinger的博客 241

【Python爬虫(1)】专栏开篇:夯实Python基础

要想熟练掌握 Python 爬虫技术,扎实的 Python 基础是必不可少的。这就好比建造高楼大厦,Python 基础就是那稳固的基石。只有打好了这个基础,我们才能在爬虫的学习道路上稳步前行,理解并运用各种爬虫技术和框架。

邓邓子的博客 2026

linux 基于debian/ubuntu AB系统适配(二)-updateEngine移植

其实A/B系统的原理很简单,大部分的工作都集中在uboot上,rootfs只是起到一个辅助的作用,但这个作用也就必不可少的。你会发现在这个其中并没有kernel的参与。

badbayyj的博客 1036

霍尼韦尔SoftMaster使用说明

霍尼韦尔SoftMaster使用说明

TMS_WebGMaps_1.9.8.0_Full_Source

TMS_WebGMaps_1.9.8.0_Full_Source 全源码 delphi cbuilder

温度PID控制PLC程序

温度PID稳定控制程序,适用于欧姆龙PLC,对其他PLC也具有很大参考价值

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

上一篇: 2025_NIPS_Retro-R1: LLM-based Agentic Retrosynthesis
下一篇: 2025_NIPS_PANGEA: Projection-Based Augmentation with Non-Relevant General Data for Enhanced Domai...
UnknownBody
博客等级 码龄16年 5601粉丝 4754原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

UnknownBody

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值