SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba

这篇文章聚焦于解决大语言模型(LLMs)高能耗问题,提出了基于脉冲神经网络(SNN)的SpikingMamba模型,核心是通过知识蒸馏从Mamba模型迁移能力,在大幅提升能效的同时最小化精度损失。
在这里插入图片描述

一、文章主要内容总结

  1. 研究背景

    • 现有LLMs(包括Mamba)依赖密集矩阵运算,能耗高,难以部署在边缘设备;SNN虽通过稀疏脉冲降低能耗,但现有SNN-based LLMs存在“精度-能效”权衡,且全量预训练成本极高。
    • 对比量化与SNN两种能效优化方案:量化无法消除密集矩阵运算的固有成本,而SNN通过二进制脉冲的时间和通道稀疏性,能进一步缓解边缘设备的I/O瓶颈。
  2. 核心方案:SpikingMamba模型

    • 基于预训练Mamba模型,通过单阶段知识蒸馏迁移能力,无需全量预训练;同时引入强化学习(DPO/KTO)进一步提升精度。
    • 关键组件包括:三元整数脉冲神经元(TI-LIF)、训练专属的平滑梯度补偿路径(SGC),以及“蒸馏+RL”的混合训练框架。
  3. 实验结果

    • 1.3B参数的SpikingMamba相比原Mamba,能效提升
船载 OpenHarmony AIS 航海工作台:告警与防碰撞预警系统 船载OpenHarmony AIS航海工作台的告警与防碰撞系统,集成十余类告警(超速、偏航、会遇、SOS等),通过网格空间索引实现高效邻近判定,保障实时性。告警管理器统一调度规则、去重、通知与日志,支持悬浮球、语音、跑马灯、弹窗多渠道触达,确保航行安全。SOS紧急通道独立运行,避免干扰。系统设计强调性能优化与多场景适配,为航海安全提供核心支撑。 阅读详情

相关推荐

AI 应用开发:四维架构落地实战——以 HR Agent 为例拆解感知·推理·记忆·执行【意图路由、规划编排、长期记忆、工具执行、面试题】

本文基于四维架构(感知·推理·记忆·执行),以HR Agent为例,系统拆解大模型应用从“聊天”到“生产可用”的落地路径。通过Spring AI实现意图路由、规划编排、RAG检索与工具调用,结合真实踩坑复盘,揭示架构化设计对安全、成本与可维护性的关键价值,为AI Agent工程化提供可复用的实战框架。

王大师企业官方博客 733

HFSS微带线仿真从零到一:材料参数与求解设置优化指南

在射频电路设计中,微带线仿真涉及电磁场求解与传输线理论,其精度高度依赖材料参数和求解设置。相对介电常数、损耗角正切和导体电导率等参数直接影响特性阻抗与插入损耗,而求解频率、网格划分和参数扫描则决定仿真效率与收敛性。合理设置这些参数可显著提升S参数准确性,避免设计迭代。应用场景包括天线馈线、滤波器和微波互连。本文基于HFSS微带线仿真实践,系统讲解材料选择与参数优化全链路,帮助工程师快速复现可靠模型。

weixin_29192211的博客 81

SD - WAN:企业出海的网络加速器

例如,一家中国的跨境电商企业,在欧美市场的促销活动期间,借助 SD - WAN 技术,即使面对海量订单数据的交互,也能确保欧美客户在浏览商品、下单支付时感受到近乎本地网络的流畅体验,大大提升了客户满意度,稳住了销售增速。企业无需在海外本地部署大量冗余服务器,只需依托云台,借助 SD - WAN 快速稳定的网络连接,即可轻松实现业务拓展,进一步精简运营架构,提升整体效能。在未来,随着技术的不断演进和企业出海需求的持续增长,SD - WAN 将持续赋能,助力更多企业在全球市场扬帆远航,书写辉煌篇章。

idcicp_com的博客 448

基于深度学习的UWB门禁检测

针对基于UWB技术智能门禁管理系统,由于测距误差或非视距原因导致门禁产生不合理动作的现象。分析如何采用基于深度学习的方法,通过对TWR测距数据、TWR监听数据、到达角度差(PDOA)、信道脉冲响应(CIR)等数据维度进行深度学习,利用 TensorFlow建立相应的推导模型,对卡片姿态进行更加精准可靠的检测,从而更好完成闸机开合控制。

tenth_quadrant的博客 2027

不只是抓取:用五指灵巧手(ROH-A001/RH56)玩转ROS2下的力控与混合控制

本文深入探讨了在ROS2环境下使用五指灵巧手(ROH-A001/RH56)实现力控与混合控制的实战技巧。从硬件通信协议解析到ROS2节点开发,详细介绍了力感知架构、实时数据处理和混合控制策略,帮助开发者在Ubuntu系统中高效实现精密操作。通过鸡蛋抓取等实例,展示了力控技术在机器人灵巧操作中的关键作用。

weixin_28221501的博客 287

ShardingSphere-JDBC-若依框架集成(SpringBoot)

ShardingSphere基础知识、ShardingSphere-JDBC如何集成进若依框架中 使用的是若依框架前后端版本、动态数据源,可自行切换,默认数据源为达梦8代理端,

编程小菜鸟的笔记 8168

Android 10.0 SystemUI 屏蔽状态栏消息推送弹出的悬浮通知

在状态栏中,有一些通知类似 消息推送 弹出的 悬浮通知,弹出30秒之后会消失掉 要想屏蔽掉这些通知,首选知道这些通知是怎么产生的 通知类型 NotificationContentInflater.java 详细的介绍各种通知的类型 frameworks/base/packages/SystemUI/src/com/android/systemui/statusbar/notification/row/NotificationContentInflater.java /** * The defau

安卓兼职framework和app工程师的博客 1542

Android实现TextToSpeech控件实现语音播报--朗读文字 (附完整源码)

Android实现TextToSpeech控件实现语音播报--朗读文字 (附完整源码)

希望我的博客,能帮上你解决学习中工作中所遇到的问题 1121

Open3D 计算点云法向量的夹角【2026最新版】

计算法向量的夹角,博客长期更新,本文最新更新时间为:2026年3月7日。

点云侠的博客 3131

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

jre9下载 jdk9

(Java SE Runtime Environment 9.0) 9.0u175 官方最新版 【x86|x64】

数学建模模型总结

这是本人在两年内准备参加数学建模竞赛搜集到的模型资料,已经经过了本人的筛选和整理,希望能够对准备参加数学建模竞赛的学弟学妹们提供一些帮助!

RU DOS版、EFI版32bit/64bit BIOS常用工具

DOS版、EFI版的RU工具,可用来查看IO、MEM、PCI、SMBIOS等资源。亲测在EFI SHELL下可用。

上一篇: UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
下一篇: FedSRD: Sparsify-Reconstruct-Decompose for Communication-Efficient Federated Large Language Models
UnknownBody
博客等级 码龄16年 5601粉丝 4740原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

UnknownBody

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值