Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large La...

文章总结与翻译

一、主要内容

本文提出了一种名为Blu-WERP的新型数据预处理管道,专门用于优化Common Crawl WARC文件的质量,以适配大型语言模型(LLM)训练。该管道通过文本提取、去重、分类器过滤三大核心模块,解决了现有预处理方案在大规模数据清洗、语义相关性保留与下游模型性能平衡方面的不足。

研究团队在150M至1B参数的多种模型上,基于9个标准基准(涵盖世界知识与推理、语言理解、常识推理三大类别)进行了全面评估。结果显示,Blu-WERP在所有模型规模下均表现优异:在1B参数规模时,相较于DCLM和Fineweb分别实现了4.0%和9.5%的综合性能提升,同时提升了每token的质量效率;在分类别评估中,世界知识与推理任务提升2.4%,语言理解任务提升6.2%,常识推理任务提升4.2%。此外,通过基于缩放定律的数据集选择协议,该管道在多十亿参数模型上的性能优势也得到了预测验证。

二、创新点

  1. 语义感知与基准驱动的过滤机制:融合FastText分类器与BETR(基准目标排序)方法,实现动态反馈式过滤,优先保留语义丰富且与基准任务高度对齐的内容,突破了传统静态启发式规则的局限性。
  2. 分层去重策略:采用优化后的Bloom Filter分层去重方案(支持段落级与文档级联动检测),在降低内存占用的同时,有效去除精确重复与近重复内容,平衡了计算效率与去重准确性。
  3. 多模块协同优化:通过系统的消融实验,筛选出最优文本提取器(Justex
船载 OpenHarmony AIS 航海工作台:告警与防碰撞预警系统 船载OpenHarmony AIS航海工作台的告警与防碰撞系统,集成十余类告警(超速、偏航、会遇、SOS等),通过网格空间索引实现高效邻近判定,保障实时性。告警管理器统一调度规则、去重、通知与日志,支持悬浮球、语音、跑马灯、弹窗多渠道触达,确保航行安全。SOS紧急通道独立运行,避免干扰。系统设计强调性能优化与多场景适配,为航海安全提供核心支撑。 阅读详情

相关推荐

AI 应用开发:四维架构落地实战——以 HR Agent 为例拆解感知·推理·记忆·执行【意图路由、规划编排、长期记忆、工具执行、面试题】

本文基于四维架构(感知·推理·记忆·执行),以HR Agent为例,系统拆解大模型应用从“聊天”到“生产可用”的落地路径。通过Spring AI实现意图路由、规划编排、RAG检索与工具调用,结合真实踩坑复盘,揭示架构化设计对安全、成本与可维护性的关键价值,为AI Agent工程化提供可复用的实战框架。

王大师企业官方博客 1302

从零搭建私有AI工作站:Ollama+DeepSeek+Dify本地化部署全攻略

本文提供了一份详尽的本地化AI工作站搭建指南,核心在于实现数据安全与离线可用。通过整合Ollama、DeepSeek与Dify三大工具,手把手教你完成从环境配置、模型部署到可视化应用构建的全过程,重点解决了Dify连接Ollama等关键步骤,助你打造一个完全自主可控的私有AI开发环境。

weixin_28396629的博客 314

SAP PP模块 MRP配置实战:5步完成工厂级物料需求计划激活

本文详细介绍了SAP PP模块中物料需求计划(MRP)的工厂级配置实战,通过5个关键步骤从基础环境准备到最终激活。内容涵盖SPRO后台配置、MRP控制者定义、主数据设置及核心参数调整,帮助顾问快速掌握MRP配置清单的完整流程,确保物料需求计划的高效运行。

weixin_30431445的博客 346

深入Android构建系统:一次修改ro.vendor.build.fingerprint失败的排查与解决记录

本文深入探讨了Android构建系统中修改ro.vendor.build.fingerprint的完整解决方案。通过分析vendor分区的特殊地位和构建指纹的生成路径,提供了三种实战修改方法,并揭示了指纹修改可能遇到的陷阱与对策,帮助开发者彻底解决这一常见问题。

weixin_30621711的博客 144

Linux系统遭遇挖矿病毒深度排查与根治指南

本文提供了一份详尽的Linux服务器挖矿病毒排查与根治指南。当CPU异常满载但常规top命令无法定位进程时,需警惕病毒通过ld.so.preload文件隐藏。文章逐步指导如何清除恶意文件、进程、定时任务,并重点应对pamdicks等顽固变种,最后给出深度排查与系统加固方案,确保服务器安全。

weixin_29207533的博客 454

算法系列之十二:多边形区域填充算法--几种边标志填充算法

四、边界标志填充算法 在光栅显示平面上,多边形是封闭的,它是用某一边界色围成的一个闭合区域,填充是逐行进行的,即用扫描线逐行对多边形求交,在交点对之间填充。边界标志填充算法就是在逐行处理时,利用边界或边界颜色作为标志来进行填充的。准确地说,边界标志填充算法不是指某种具体的填充算法,而是一类利用扫描线连贯性思想的填充算法的总称。这类算法有很多种,本篇就介绍几种。 首先介绍一种以边为中心的边缘填充...

java开发指南博客 【转载】 1898

Matlab实战:手把手教你用Zernike多项式模拟光学像差(附完整代码)

本文详细介绍了如何在Matlab中使用Zernike多项式模拟光学像差,包括基础理论、不同排序系统的实现、完整代码示例以及高级应用技巧。通过Zernike多项式,可以有效地描述和分析光学系统中的波前畸变,适用于望远镜校正、眼科光学等多个领域。

weixin_42523720的博客 162

SpringBoot+Vue实现图片上传

SpringBoot+Vue实现图片上传

叶♥璃 2736

2025最新!视频动作识别算法的前沿研究与论文解读

2025最新!视频动作识别算法的前沿研究与论文解读 ,人工智能,计算机视觉,大模型,AI,本文聚焦2025年视频动作识别算法前沿,先介绍其定义、意义及核心挑战,再阐述传统与深度学习方法。还解读了基于Transformer、轻量化、跨模态融合和半监督学习的前沿研究,解释关键概念,列举常用数据集、评价指标,分析当前问题与未来趋势,为该领域研究提供全面参考。

不踩坑,不知道坑有多深;不总结,踩过的坑白踩。 7万+

Probability and Statistics - 4th edition

Probability and Statistics - 4th edition

visio安装包

Microsoft Office Visio 是一款专业的制图软件。利用经过更新的形状、协作工具和数据链接图表,创建专业级图表以简化复杂的信息。

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

RU DOS版、EFI版32bit/64bit BIOS常用工具

DOS版、EFI版的RU工具,可用来查看IO、MEM、PCI、SMBIOS等资源。亲测在EFI SHELL下可用。

上一篇: The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmi...
下一篇: Active Slice Discovery in Large Language Models
UnknownBody
博客等级 码龄16年 5601粉丝 4754原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

UnknownBody

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值