GBDT和deep learning之间对比和争论

DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks 论文链接:https://dl.acm.org/citation.cfm?id=3330858 代码链接:https://github.com/motefly/DeepGBM 背景 Guolin Ke是lightgbm的核心作者之一,我幸... 阅读详情

Boosted Decision Trees for Deep Learning

Tags:  Deep,  Machine Learning,  Supervised —  jl@ 11:18 am

About 4 years ago, I speculated that decision trees qualify as a deep learning algorithm because they can make decisions which are substantially nonlinear in the input representation. Ping Li has proved this correct, empirically at UAI by showing that boosted decision trees can beat deep belief networks on versions of Mnist which are artificially hardened so as to make them solvable only by deep learning algorithms.

This is an important point, because the ability to solve these sorts of problems is probably the best objective definition of a deep learning algorithm we have. I’m not that surprised. In my experience, if you can accept the computational drawbacks of a boosted decision tree, they can achieve pretty good performance.

Geoff Hinton once told me that the great thing about deep belief networks is that they work. I understand that Ping had very substantial difficulty in getting this published, so I hope some reviewers step up to the standard of valuing what works.

10 Comments to “Boosted Decision Trees for Deep Learning”
  1. After seeing Ping’s talk about this work in February, I spent some time reproducing his results. Getting the same results as Ping turned out to be fairly easy. It certainly required a lot less tweaking than reproducing the results of DBNs and the like on the same data sets.

    Having said that, I am not convinced that deep learning does a good job on all hardened MNIST data sets. For instance, on the data set that has natural images as background for the digits, today’s deep learners almost have to do a bad job: the unsupervised pre-training algorithms are wasting a lot of modeling power on trying to model the background images, the structure of which is completely irrelevant for the classification task. The subsequent discriminative fine-tuning then has to fix this, but can only do so much. So on some of the data sets considered in the paper, I don’t think it is too surprising that powerful discriminative learners such as boosted regression trees can do a better job than deep learning.

  2. Itman  says:

    Quite interesting. I know that the Russian search engine Yandex uses boosted decision trees (similar to TreeRank) to implement a learning to rank approach.

  3. Yoshua Bengio  says:

    Decision trees actually are shallow learners (2-levels, e.g. like SVMs and ordinary MLPs), as discussed in a paper soon to appear (but already available online there:http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/436). Basically a decision tree can be written as a sum-of-products where each product (over the arcs from the root to a leaf) selects a leaf. The paper discusses how they can only generalize locally (the training examples within each region defined by a leaf only provide examples about test examples in the same region). On the other hand, boosted decision trees add one level of depth to the weak learners, which makes them reasonably deep (considering that many deep learning results are often with just 3 or 4 levels).

  4. hal  says:

    @Yoshua: I’m afraid I don’t quite follow something. You’re saying that trees are sums of products (eq 1 in the paper), which I totally buy. So those are two layers, and when you add boosting, you get three. This is stated in the paper as:

    “What is the architectural depth of decision trees and decision forests? It depends on what elementary units of
    computation are allowed on each level. By analogy with the disjunctive normal form (which is usually assigned an
    architectural depth of two) one would assign an architectural depth of two to a decision tree, and of three to decision
    forests or boosted trees.” (page 10)

    I guess I don’t completely understand the reasoning behind saying that DNF has depth 2. Is there a citation I could look at that argues this point?

    To me, it seems like you could make this claim about almost any learning algorithm. The argument for trees reads to me like “trees are just sums of [things]“, where [things] = a bunch of products over features. But couldn’t I say this about other algorithms? For instance, take a depth 1000 neural network. Why couldn’t I say that this is just two layers because the final layer is a sum of [things], where in this case [things] is another non-linear function over the features? Why do products not get to count as layers, but thresholded sums do get to count as layers?

    (I’m not trying to be cantankerous — I’m really trying to understand!)

    • Yoshua Bengio  says:

      Hal,

      Yann LeCun and I wrote a paper on depth (Scaling learning algorithms toward AI: http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/4) in which we discuss and define depth and give many examples. To define the depth of an architecture requires to also define what are the elementary computational units of the graph (whose depth is the longest path from input to output node). If we change the definition of what these units can do, we get a different value for depth, but such redefinitions typically only change depth by a multiplicative constant (e.g. we can replace a formal neuron by the composition of three kinds of things, sums, products and thresholds, i.e., multiply depth of a neural net by 3 by changing what we consider to be the elementary units of computation). In computational complexity papers, the typical elementary units that have been considered are summing units, product units, linear threshold units (formal neuron), and logic gates. Hence in the above paper, a linear classifier is typically considered to have depth 1, whereas a finite polynomial (sum of products) is considered to have depth 2, a traditional MLP (with one hidden layer) is considered to have depth 2, and an SVM or RBF network or kernel-based predictor is considered to have depth 2 (either the feature function phi(x) or the kernel function K(u,v)=phi(u).phi(v) can be considered in the set of computational elements). Nicely, we find that with all these kinds of computational element sets, depth 1 is insufficient to represent most functions, whereas depth 2 provides a universal approximator. Note how that works also for decision trees (which can basically be decomposed as a sum of products), that according this cultural convention also have depth 2 and are universal approximators. Boosting on top of decision trees adds a level of depth (it is a weighted sum of the hard decisions produced by the decision trees). That is why I wrote that that boosted decision trees have depth 3.

      In the paper on Scaling learning algorithms toward AI, we also introduce the notion that for learned architectures, we typically would like each level in the architecture to be adapted. Hence a depth 1000 neural net is a weighted sum over the outputs of the 999-th layer, but all 1000 layers are adaptive, and composed of the same kind of computational units, so it makes more sense to talk about a depth 1000 architecture in that case. Note how when we learn a polynomial we typically can adapt the coefficients (2nd layer parameters) and choose which products enter the polynomial (1st layer parameters), although of course one can consider all the possible products and then the 1st layer is not adaptive anymore, only the second. But practically, we generally can’t afford the coefficients of all possible products to be non-zero, so we need to do some kind of ‘feature selection’, which amounts to learning the first layer.

      Let me know if my explanations answer your questions and satisfy you. In conclusion, there is a formal definition of depth which can give different answers in practice because it depends on the choice of computational units, and there is a softer more fuzzy definition of depth that represents a kind of cultural convention (for computational elements) that typically makes depth-2 architectures to be universal approximators (with enough units in the first layer). It is under this convention that decision trees would reasonably be seen as depth-2 and boosted decision trees as depth-3 architectures.

Python-DeepGBMGBDT针对在线预测任务发布的深度学习框架 DeepGBM:GBDT针对在线预测任务发布的深度学习框架 立即下载

相关推荐

LightGBM中的GBDT实现

LightGBM中的GBDT实现

新型在线预测模型DeepGBM(基于GBDT扩展的深度学习框架)

DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction TasksGuoli...

hestendelin的专栏 4310

iOS马甲包过审实战:如何用5天时间搞定4.3审核(附完整代码混淆方案)

本文提供了一套为期五天的高效实战方案,专门解决iOS应用因重复或抄袭问题而被App Store的4.3条款拒绝的难题。方案涵盖深度分析、代码混淆、资源翻新及环境隔离等系统性步骤,并附有完整的代码混淆示例,旨在帮助开发者在合规框架内重塑应用,显著提升过审成功率。

weixin_29234239的博客 148

推荐系统遇上深度学习(十)--GBDT+LR融合方案实战

推荐系统遇上深度学习系列: 推荐系统遇上深度学习(一)--FM模型理论和实践:https://www.jianshu.com/p/152ae633fb00 推荐系统遇上深度学习(二)--FFM模型理论和实践:https://www.jianshu.com/p/781cde3d5f3d 推荐系统遇上深度学习(三)--DeepFM模型理论和实践:https://www.jianshu.com/p/6f1c2643d31b 推荐系统遇上深度学习(四)--多值离散特征的embedding解决方案:https://w

704

智能变电站协议系列-1、GOOSE、SV、MMS协议简介及GOOSE示例运行问题(IEC61850)

目前有一个新的研究,关于电网使用的GOOSE、SV、MMS协议,并使用相关库来做一下测试。由于goose、sv、mms等都属于IEC61850,因此,需要对IEC61850相关内容有一些了解,都是国际标准,属于电气自动化的一部分,对于自动化或者电气相关领域有接触的专业应该或多或少有一些了解。IEC61850目标是用于变电站自动化、分布式能源、水力和天然气发电厂、风力发电厂、电动汽车作为分布式能源等领域,目前似乎在变电站自动化方面的推广和使用更加广泛。

逍遥游的博客 2万+

从SPI协议到文件系统:手把手教你用HAL库给W25Q16移植FATFS(附扇区管理优化方案)

本文详细介绍了如何在STM32平台上,通过HAL库的SPI接口为W25Q16 Flash芯片移植FATFS文件系统。文章从SPI通信基础讲起,深入解析了FATFS底层驱动(diskio.c)的移植要点,并重点探讨了针对W25Q16的4KB物理扇区特性进行扇区管理优化的两种方案,旨在帮助开发者构建稳定高效的文件存储方案。

happy2的博客 1046

【机器学习】为什么GBDT可以超越深度学习

今天的讨论比较玄学,至于为什么用曹雪芹的诗当封面,看完文章明白啦。目录背景玄学立论娓娓道来后话背景之前在知乎上回答了一个类似的问题,他们问为什么很多领域的比赛数据集上的GBDT效果那么好(...

fengdu78的博客 744

海康工业相机触发模式详解:软触发 vs 外触发,在C#里如何选与用?

本文深入解析海康工业相机的软触发与外触发模式,详细比较两者的特性差异、适用场景及C#实现方法。通过海康SDK配置指南和实战代码示例,帮助开发者掌握工业相机触发模式的选择与优化技巧,提升自动化检测系统的精确性和可靠性。

weixin_42635064的博客 557

数据挖掘中的利器--XGBoost理论篇

XGBoost是各种数据挖掘或机器学习算法类比赛中每个团队都会使用且精度相对最好的算法之一(Deep Learning算法除外)。也就是说,对于刚转向机器学习领域的同胞们,在掌握数据挖掘的基本常识概念之后,要想在比赛中有所收获,掌握XGBoost算法也是当务之急。1、XGBoost算法优点  XGBoost 是 Extreme Gradient Boosting的简称。它是Gradient Boo

学习AI算法,请关注微信公众号:机器学习算法全栈工程师…… 7093

10分钟搞定小红书爆款笔记爬取!Selenium模拟登录+数据解析全流程,新手也能落地

小红书爬虫实战的核心,是**“模拟真人操作+解析结构化数据”**——Selenium解决了动态页面和登录的问题,lxml解决了数据提取的问题,pandas解决了数据存储的问题。新手学爬虫不用一开始就啃复杂的反爬机制,先从这类“可见即可爬”的场景入手,掌握“模拟操作→数据解析→数据保存”的全流程,再逐步叠加异常处理、批量爬取、反爬应对等功能。需要注意的是:爬取数据仅用于个人学习和研究,不得用于商业用途,且需遵守小红书的用户协议,不要过度爬取导致服务器压力。

专注于Python爬虫开发,分享爬虫技巧、项目实战与反爬经验,使用Scrapy、BeautifulSoup等工具,解决数据抓取难题。 1730

中文核心:swin transformer 农作物病虫害识别 完整代码数据

中文核心:swin transformer 农作物病虫害识别

942

论文笔记:《DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks》

论文笔记:《DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks》摘要1. 简介2. DEEPGBM2.1 CatNN 专注于稀疏类别特征(for Sparse Categorical Features)2.2 GBDT2NN 专注于数值密集型特征(for Dense Numerical ...

fuzi2012的博客 5417

五万字解析【ROOT 入门指南】

本文是cern官方root入门指南的初步翻译版,还在进一步校对中....

荷塘阅色 210

#Paper Reading# DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks

论文题目: DeepGBM: A Deep Learning Framework Distilled by GBDT for Online Prediction Tasks 论文地址: https://dl.acm.org/citation.cfm?id=3330858 论文发表于: KDD 2019 论文大体内容: 本文主要提出了DeepGBM模型,来整合NN与GBDT的优势,支持sparse...

John159151的专栏 744

YOLO26涨点改进| TGRS 2026 | 独家创新首发、注意力改进篇| 引入LCGA局部曲率引导注意力,保持局部稳定性的同时突出关键几何区域,含多种创新改进,促进YOLO26所有任务高效涨点

🔥本文提出LCGA(局部曲率引导注意力)模块,通过引入曲率信息改进YOLO26网络模型。LCGA在局部窗口内利用深度卷积构造曲率代理,调制标准窗口注意力,使模型更关注道路、边界等曲线型结构,提升遥感图像超分辨率中的几何保真度。相比普通注意力,LCGA能精准聚焦关键几何区域,减少边缘断裂和轮廓偏移,同时保持较低计算开销。实验表明LCGA显著提升性能,在UCMerced数据集上PSNR提升0.2148dB。文章提供了三种改进方案。

Ai缝合怪、助力小伙伴高效发论文 228

计算机控制技术及应用

《计算机控制技术及应用》通过总结作者的科研成果和吸收国内外的先进理论、方法和技术,以计算机控制系统的网络化、开放化、智能化和集成化发展趋势为主线索,主要介绍计算机控制系统的基本概念和计算机控制系统的典型形式,计算机控制系统的数学描述与基本分析方法,以及数字PID控制器、最少拍控制系统、达林算法、状态变量设计法等设计方法及其各种改进算法。重点突出计算机控制系统输入输出通道接口的设计方法和典型控制系统的设计案例与实现方法等。

推荐系统第四课(用户建模----召回排序都会用到)wide and deep Practice

1 learning to rank 回顾 Learning to Rank for Information Retrieval,Tie-Yan :Liu https://www.cda.cn/uploadfile/image/20151220/20151220115436_46293.pdf 2、点击预估模型 模型:

瑶子的博客 4819

android 蓝牙通讯demo

android 下蓝牙通讯demo,实现了发现蓝牙,使能蓝牙,通过蓝牙发送消息。

Wide & Deep Learning for Recommender Systems阅读笔记

目录一、论文背景介绍1. CRT预估2. 推荐模型的特点二、本篇文章1. 引言2. 内容3. 贡献三、Model1. Wide 模型2. Deep 模型3. 联合训练联合训练和集成训练的区别:四、系统实现1. 数据生成2. 模型训练3. 线上应用五、实验总结 一、论文背景介绍 我们今天讲的论文属于框架中的排序层。 1. CRT预估 CTR预估(对每次广告的点击情况做出预测:点击还是不点击)可以广泛应用于个性化推荐、信息检索、在线广告等领域,是互联网广告中最核心的算法之一,预估的准确性与否直接直接影响

weixin_44853181的博客 716

廖雪峰Python教程[完整版]

廖雪峰的python教程整理的,已经排好版了,而且还是完整的,适合小白

lwip-2.0.3

lwip是瑞典计算机科学院(SICS)的Adam Dunkels 开发的一个小型开源的TCP/IP协议栈。实现的重点是在保持TCP协议主要功能的基础上减少对RAM 的占用。

#Paper Reading# Wide & Deep Learning for Recommender Systems

论文题目: Wide & Deep Learning for Recommender Systems 论文地址: https://dl.acm.org/citation.cfm?id=2988454 论文发表于: DLRS 2016(RecSys 2016的Workshop,CORE B类会议) 论文大体内容: 本文主要提出了WDL(Wide & Deep Learning)模型...

John159151的专栏 652

【阅读笔记】Applying Deep Learning To Airbnb Search

Applying Deep Learning To Airbnb Search Airbnb Inc. malay.haldar@airbnb.com ABSTRACT 最初使用 gradient boosted decision tree model 来做 search ranking ,搜索效果从刚开始的上升逐渐趋于稳定。本文讨论如何突破趋于平稳的效果。本文的目的不是讲述模型上的突破,而是如何...

SrdLaplace的博客 2290

applying deep learning to airbnb search

本文原发于公众号:机器学习与推荐系统 这周分享一篇文章:Applying Deep Learning To Airbnb Search,内容如论文标题所示,介绍了爱彼迎如何在自己的搜索工程中应用深度学习。 这篇论文有趣的地方在于,并不是提出一个新颖的算法或模型,而是介绍了爱彼迎在应用深度学习的过程中,发现了哪些有趣且有用的点,进而分享给大家。在此之前,作者表示一直使用 GBDT 进行搜索排序,然而已经陷入了一个瓶颈期,需要深度学习来帮助打破这个瓶颈。 接下来我们一起看看,爱彼迎在应用深度学习的...

默默学习 440

Applying Deep Learning To Airbnb Search简介

Applying Deep Learning To Airbnb Search Introduction 首先,作者介绍了airbnb的业务流程:从客人在airbnb上搜索合适的房子,系统给出具有一定顺序的列表房间。 同时介绍了airbnb的技术变迁,从最早的人力手动打分,后面又使用gbdt模型有了很大的提高,再是顺应时代变迁在神经网络有持续性的探索,最后收获了满意的结果。 搜索排序模型的主要生态...

loserChen的博客 620

2019表格学习_NEURAL OBLIVIOUS DECISION ENSEMBLES FOR DEEP LEARNING ON TABULAR DATA

如今,深度神经网络 (DNN) 已成为视觉、NLP 和语音等广泛领域机器学习任务的主要工具。与此同时,在异构表格数据的一个重要案例中,DNN 相对于浅层网络的优势仍然值得怀疑。特别是,没有足够的证据表明深度学习机制允许构建优于梯度提升决策树(GBDT)的方法,而梯度提升决策树通常是表格问题的首选。在本文中,我们介绍了神经不经意决策集成(NODE),这是一种新的深度学习架构,旨在处理任何表格数据。

qq_45972323的博客 1100

论文研读系列——“Revisiting Deep Learning Models for Tabular Data”

Revisiting Deep Learning Models for Tabular Data arxiv [Submitted on 22 Jun 2021 , last revised 26 Oct 2023 ] 链接:arXiv:2106.11959 代码:https://github.com/Yura52/tabular-dl-revisiting-models https://github.com/yandex-research/tabular-dl-revisiting-models 摘要 本

Water8L的博客 3145

论文笔记|Wide & Deep Learning for Recommender Systems

在读了FM和FMM的论文后,紧接着开始学习最典型的FM系列的深度神经网络模型DeepFM,看到论文中多次提到Wide&Deep这个模型,因此首先找来这篇文章学习了一下。 通常分类或回归方法分为两类,一类是简单的线性模型,通过分析特征性质,用两个或多个特征手动构建新的特征,特点是输入变量高维、稀疏,称为Wide模型,如LR;另一类是模型较为复杂,能够自动构建新的非线性特征,,特点是输入变量低...

年迈的艺术家们 434

2023数据嵌入_On Embeddings for Numerical Features in Tabular Deep Learning

最近,类似 Transformer 的深度架构在表格数据问题上表现出了强大的性能。与 MLP 等传统模型不同,这些架构将数值特征的标量值映射到高维嵌入,然后再将它们混合到主干中。在这项工作中,我们认为数字特征的嵌入是表格 DL 中尚未充分探索的自由度,它允许构建更强大的 DL 模型,并在一些 GBDT 友好的基准上与梯度增强决策树 (GBDT) 竞争(即,GBDT 优于传统的深度学习模型)。我们首先描述两种概念上不同的构建嵌入模块的方法:第一种方法基于标量值的分段线性编码,第二种方法利用周期性激活。

qq_45972323的博客 807

GBDT,LR,深度学习如何联合发力(以推荐系统和游戏为例)

GBDT, LR & Deep Learning for Turn-based Strategy Game AILike Zhang, Hui Pan, Qi F...

hestendelin的专栏 596
上一篇: win7删除文件时,提示“操作无法完成,因为其中的文件夹或文件已在另一程序中打开”问题
下一篇: sort使用tab键作为分隔符,出现“sort: multi-character tab”错误
overstack
博客等级 码龄14年 115粉丝 55原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值