Spark Gradient-boosted trees (GBTs)梯度提升树

最新推荐文章于 2024-11-18 17:27:05 发布
最新推荐文章于 2024-11-18 17:27:05 发布
阅读量102 收藏
点赞数
CC 4.0 BY-SA版权
文章标签： 大数据 数据结构与算法
原文链接：http://www.cnblogs.com/xiaoma0529/p/7230297.html
本文介绍了梯度提升树（GBT）在Spark MLlib中的实现及其在回归任务的应用。通过具体实例展示了如何导入相关包、准备数据、构建GBT模型、训练模型以及评估模型性能。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

梯度提升树（GBT）是决策树的集合。 GBT迭代地训练决策树以便使损失函数最小化。 spark.ml实现支持GBT用于二进制分类和回归，可以使用连续和分类特征。

 

GBDT的优点

 

　　GBDT和随机森林一样，都具备决策树的一些优点： 
　　(1)可以处理类别特征和连续特征； 
　　(2)不需要对数据进行标准化预处理； 
　　(3)可以分析特征之间的相互影响 
　　值得注意的是，Spark中的GBDT目前还不能处理多分类问题，仅可以用于二分类和回归问题。（Spark随机森林可以处理多分类问题）　

 

导入包

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Row
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.Column
import org.apache.spark.sql.DataFrameReader
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.Encoder
import org.apache.spark.sql.DataFrameStatFunctions
import org.apache.spark.sql.functions._
 
import org.apache.spark.ml.linalg.Vectors
import org.apache.spark.ml.feature.{ IndexToString, StringIndexer, VectorIndexer }
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.regression.{ GBTRegressionModel, GBTRegressor }
import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.tuning.{ ParamGridBuilder, CrossValidator }

　　导入数据源

val spark = SparkSession.builder().appName("Spark Gradient-boosted tree regression").config("spark.some.config.option", "some-value").getOrCreate()
 
// For implicit conversions like converting RDDs to DataFrames
import spark.implicits._
 
val dataList: List[(Double, String, Double, Double, String, Double, Double, Double, Double)] = List( 
      (0, "male", 37, 10, "no", 3, 18, 7, 4), 
      (0, "female", 27, 4, "no", 4, 14, 6, 4), 
      (0, "female", 32, 15, "yes", 1, 12, 1, 4), 
      (0, "male", 57, 15, "yes", 5, 18, 6, 5), 
      (0, "male", 22, 0.75, "no", 2, 17, 6, 3), 
      (0, "female", 32, 1.5, "no", 2, 17, 5, 5), 
      (0, "female", 22, 0.75, "no", 2, 12, 1, 3), 
      (0, "male", 57, 15, "yes", 2, 14, 4, 4), 
      (0, "female", 32, 15, "yes", 4, 16, 1, 2), 
      (0, "male", 22, 1.5, "no", 4, 14, 4, 5), 
      (0, "male", 37, 15, "yes", 2, 20, 7, 2), 
      (0, "male", 27, 4, "yes", 4, 18, 6, 4), 
      (0, "male", 47, 15, "yes", 5, 17, 6, 4), 
      (0, "female", 22, 1.5, "no", 2, 17, 5, 4), 
      (0, "female", 27, 4, "no", 4, 14, 5, 4), 
      (0, "female", 37, 15, "yes", 1, 17, 5, 5), 
      (0, "female", 37, 15, "yes", 2, 18, 4, 3), 
      (0, "female", 22, 0.75, "no", 3, 16, 5, 4), 
      (0, "female", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "female", 27, 10, "yes", 2, 14, 1, 5), 
      (0, "female", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "female", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "female", 27, 10, "yes", 4, 16, 5, 4), 
      (0, "female", 32, 10, "yes", 3, 14, 1, 5), 
      (0, "male", 37, 4, "yes", 2, 20, 6, 4), 
      (0, "female", 22, 1.5, "no", 2, 18, 5, 5), 
      (0, "female", 27, 7, "no", 4, 16, 1, 5), 
      (0, "male", 42, 15, "yes", 5, 20, 6, 4), 
      (0, "male", 27, 4, "yes", 3, 16, 5, 5), 
      (0, "female", 27, 4, "yes", 3, 17, 5, 4), 
      (0, "male", 42, 15, "yes", 4, 20, 6, 3), 
      (0, "female", 22, 1.5, "no", 3, 16, 5, 5), 
      (0, "male", 27, 0.417, "no", 4, 17, 6, 4), 
      (0, "female", 42, 15, "yes", 5, 14, 5, 4), 
      (0, "male", 32, 4, "yes", 1, 18, 6, 4), 
      (0, "female", 22, 1.5, "no", 4, 16, 5, 3), 
      (0, "female", 42, 15, "yes", 3, 12, 1, 4), 
      (0, "female", 22, 4, "no", 4, 17, 5, 5), 
      (0, "male", 22, 1.5, "yes", 1, 14, 3, 5), 
      (0, "female", 22, 0.75, "no", 3, 16, 1, 5), 
      (0, "male", 32, 10, "yes", 5, 20, 6, 5), 
      (0, "male", 52, 15, "yes", 5, 18, 6, 3), 
      (0, "female", 22, 0.417, "no", 5, 14, 1, 4), 
      (0, "female", 27, 4, "yes", 2, 18, 6, 1), 
      (0, "female", 32, 7, "yes", 5, 17, 5, 3), 
      (0, "male", 22, 4, "no", 3, 16, 5, 5), 
      (0, "female", 27, 7, "yes", 4, 18, 6, 5), 
      (0, "female", 42, 15, "yes", 2, 18, 5, 4), 
      (0, "male", 27, 1.5, "yes", 4, 16, 3, 5), 
      (0, "male", 42, 15, "yes", 2, 20, 6, 4), 
      (0, "female", 22, 0.75, "no", 5, 14, 3, 5), 
      (0, "male", 32, 7, "yes", 2, 20, 6, 4), 
      (0, "male", 27, 4, "yes", 5, 20, 6, 5), 
      (0, "male", 27, 10, "yes", 4, 20, 6, 4), 
      (0, "male", 22, 4, "no", 1, 18, 5, 5), 
      (0, "female", 37, 15, "yes", 4, 14, 3, 1), 
      (0, "male", 22, 1.5, "yes", 5, 16, 4, 4), 
      (0, "female", 37, 15, "yes", 4, 17, 1, 5), 
      (0, "female", 27, 0.75, "no", 4, 17, 5, 4), 
      (0, "male", 32, 10, "yes", 4, 20, 6, 4), 
      (0, "female", 47, 15, "yes", 5, 14, 7, 2), 
      (0, "male", 37, 10, "yes", 3, 20, 6, 4), 
      (0, "female", 22, 0.75, "no", 2, 16, 5, 5), 
      (0, "male", 27, 4, "no", 2, 18, 4, 5), 
      (0, "male", 32, 7, "no", 4, 20, 6, 4), 
      (0, "male", 42, 15, "yes", 2, 17, 3, 5), 
      (0, "male", 37, 10, "yes", 4, 20, 6, 4), 
      (0, "female", 47, 15, "yes", 3, 17, 6, 5), 
      (0, "female", 22, 1.5, "no", 5, 16, 5, 5), 
      (0, "female", 27, 1.5, "no", 2, 16, 6, 4), 
      (0, "female", 27, 4, "no", 3, 17, 5, 5), 
      (0, "female", 32, 10, "yes", 5, 14, 4, 5), 
      (0, "female", 22, 0.125, "no", 2, 12, 5, 5), 
      (0, "male", 47, 15, "yes", 4, 14, 4, 3), 
      (0, "male", 32, 15, "yes", 1, 14, 5, 5), 
      (0, "male", 27, 7, "yes", 4, 16, 5, 5), 
      (0, "female", 22, 1.5, "yes", 3, 16, 5, 5), 
      (0, "male", 27, 4, "yes", 3, 17, 6, 5), 
      (0, "female", 22, 1.5, "no", 3, 16, 5, 5), 
      (0, "male", 57, 15, "yes", 2, 14, 7, 2), 
      (0, "male", 17.5, 1.5, "yes", 3, 18, 6, 5), 
      (0, "male", 57, 15, "yes", 4, 20, 6, 5), 
      (0, "female", 22, 0.75, "no", 2, 16, 3, 4), 
      (0, "male", 42, 4, "no", 4, 17, 3, 3), 
      (0, "female", 22, 1.5, "yes", 4, 12, 1, 5), 
      (0, "female", 22, 0.417, "no", 1, 17, 6, 4), 
      (0, "female", 32, 15, "yes", 4, 17, 5, 5), 
      (0, "female", 27, 1.5, "no", 3, 18, 5, 2), 
      (0, "female", 22, 1.5, "yes", 3, 14, 1, 5), 
      (0, "female", 37, 15, "yes", 3, 14, 1, 4), 
      (0, "female", 32, 15, "yes", 4, 14, 3, 4), 
      (0, "male", 37, 10, "yes", 2, 14, 5, 3), 
      (0, "male", 37, 10, "yes", 4, 16, 5, 4), 
      (0, "male", 57, 15, "yes", 5, 20, 5, 3), 
      (0, "male", 27, 0.417, "no", 1, 16, 3, 4), 
      (0, "female", 42, 15, "yes", 5, 14, 1, 5), 
      (0, "male", 57, 15, "yes", 3, 16, 6, 1), 
      (0, "male", 37, 10, "yes", 1, 16, 6, 4), 
      (0, "male", 37, 15, "yes", 3, 17, 5, 5), 
      (0, "male", 37, 15, "yes", 4, 20, 6, 5), 
      (0, "female", 27, 10, "yes", 5, 14, 1, 5), 
      (0, "male", 37, 10, "yes", 2, 18, 6, 4), 
      (0, "female", 22, 0.125, "no", 4, 12, 4, 5), 
      (0, "male", 57, 15, "yes", 5, 20, 6, 5), 
      (0, "female", 37, 15, "yes", 4, 18, 6, 4), 
      (0, "male", 22, 4, "yes", 4, 14, 6, 4), 
      (0, "male", 27, 7, "yes", 4, 18, 5, 4), 
      (0, "male", 57, 15, "yes", 4, 20, 5, 4), 
      (0, "male", 32, 15, "yes", 3, 14, 6, 3), 
      (0, "female", 22, 1.5, "no", 2, 14, 5, 4), 
      (0, "female", 32, 7, "yes", 4, 17, 1, 5), 
      (0, "female", 37, 15, "yes", 4, 17, 6, 5), 
      (0, "female", 32, 1.5, "no", 5, 18, 5, 5), 
      (0, "male", 42, 10, "yes", 5, 20, 7, 4), 
      (0, "female", 27, 7, "no", 3, 16, 5, 4), 
      (0, "male", 37, 15, "no", 4, 20, 6, 5), 
      (0, "male", 37, 15, "yes", 4, 14, 3, 2), 
      (0, "male", 32, 10, "no", 5, 18, 6, 4), 
      (0, "female", 22, 0.75, "no", 4, 16, 1, 5), 
      (0, "female", 27, 7, "yes", 4, 12, 2, 4), 
      (0, "female", 27, 7, "yes", 2, 16, 2, 5), 
      (0, "female", 42, 15, "yes", 5, 18, 5, 4), 
      (0, "male", 42, 15, "yes", 4, 17, 5, 3), 
      (0, "female", 27, 7, "yes", 2, 16, 1, 2), 
      (0, "female", 22, 1.5, "no", 3, 16, 5, 5), 
      (0, "male", 37, 15, "yes", 5, 20, 6, 5), 
      (0, "female", 22, 0.125, "no", 2, 14, 4, 5), 
      (0, "male", 27, 1.5, "no", 4, 16, 5, 5), 
      (0, "male", 32, 1.5, "no", 2, 18, 6, 5), 
      (0, "male", 27, 1.5, "no", 2, 17, 6, 5), 
      (0, "female", 27, 10, "yes", 4, 16, 1, 3), 
      (0, "male", 42, 15, "yes", 4, 18, 6, 5), 
      (0, "female", 27, 1.5, "no", 2, 16, 6, 5), 
      (0, "male", 27, 4, "no", 2, 18, 6, 3), 
      (0, "female", 32, 10, "yes", 3, 14, 5, 3), 
      (0, "female", 32, 15, "yes", 3, 18, 5, 4), 
      (0, "female", 22, 0.75, "no", 2, 18, 6, 5), 
      (0, "female", 37, 15, "yes", 2, 16, 1, 4), 
      (0, "male", 27, 4, "yes", 4, 20, 5, 5), 
      (0, "male", 27, 4, "no", 1, 20, 5, 4), 
      (0, "female", 27, 10, "yes", 2, 12, 1, 4), 
      (0, "female", 32, 15, "yes", 5, 18, 6, 4), 
      (0, "male", 27, 7, "yes", 5, 12, 5, 3), 
      (0, "male", 52, 15, "yes", 2, 18, 5, 4), 
      (0, "male", 27, 4, "no", 3, 20, 6, 3), 
      (0, "male", 37, 4, "yes", 1, 18, 5, 4), 
      (0, "male", 27, 4, "yes", 4, 14, 5, 4), 
      (0, "female", 52, 15, "yes", 5, 12, 1, 3), 
      (0, "female", 57, 15, "yes", 4, 16, 6, 4), 
      (0, "male", 27, 7, "yes", 1, 16, 5, 4), 
      (0, "male", 37, 7, "yes", 4, 20, 6, 3), 
      (0, "male", 22, 0.75, "no", 2, 14, 4, 3), 
      (0, "male", 32, 4, "yes", 2, 18, 5, 3), 
      (0, "male", 37, 15, "yes", 4, 20, 6, 3), 
      (0, "male", 22, 0.75, "yes", 2, 14, 4, 3), 
      (0, "male", 42, 15, "yes", 4, 20, 6, 3), 
      (0, "female", 52, 15, "yes", 5, 17, 1, 1), 
      (0, "female", 37, 15, "yes", 4, 14, 1, 2), 
      (0, "male", 27, 7, "yes", 4, 14, 5, 3), 
      (0, "male", 32, 4, "yes", 2, 16, 5, 5), 
      (0, "female", 27, 4, "yes", 2, 18, 6, 5), 
      (0, "female", 27, 4, "yes", 2, 18, 5, 5), 
      (0, "male", 37, 15, "yes", 5, 18, 6, 5), 
      (0, "female", 47, 15, "yes", 5, 12, 5, 4), 
      (0, "female", 32, 10, "yes", 3, 17, 1, 4), 
      (0, "female", 27, 1.5, "yes", 4, 17, 1, 2), 
      (0, "female", 57, 15, "yes", 2, 18, 5, 2), 
      (0, "female", 22, 1.5, "no", 4, 14, 5, 4), 
      (0, "male", 42, 15, "yes", 3, 14, 3, 4), 
      (0, "male", 57, 15, "yes", 4, 9, 2, 2), 
      (0, "male", 57, 15, "yes", 4, 20, 6, 5), 
      (0, "female", 22, 0.125, "no", 4, 14, 4, 5), 
      (0, "female", 32, 10, "yes", 4, 14, 1, 5), 
      (0, "female", 42, 15, "yes", 3, 18, 5, 4), 
      (0, "female", 27, 1.5, "no", 2, 18, 6, 5), 
      (0, "male", 32, 0.125, "yes", 2, 18, 5, 2), 
      (0, "female", 27, 4, "no", 3, 16, 5, 4), 
      (0, "female", 27, 10, "yes", 2, 16, 1, 4), 
      (0, "female", 32, 7, "yes", 4, 16, 1, 3), 
      (0, "female", 37, 15, "yes", 4, 14, 5, 4), 
      (0, "female", 42, 15, "yes", 5, 17, 6, 2), 
      (0, "male", 32, 1.5, "yes", 4, 14, 6, 5), 
      (0, "female", 32, 4, "yes", 3, 17, 5, 3), 
      (0, "female", 37, 7, "no", 4, 18, 5, 5), 
      (0, "female", 22, 0.417, "yes", 3, 14, 3, 5), 
      (0, "female", 27, 7, "yes", 4, 14, 1, 5), 
      (0, "male", 27, 0.75, "no", 3, 16, 5, 5), 
      (0, "male", 27, 4, "yes", 2, 20, 5, 5), 
      (0, "male", 32, 10, "yes", 4, 16, 4, 5), 
      (0, "male", 32, 15, "yes", 1, 14, 5, 5), 
      (0, "male", 22, 0.75, "no", 3, 17, 4, 5), 
      (0, "female", 27, 7, "yes", 4, 17, 1, 4), 
      (0, "male", 27, 0.417, "yes", 4, 20, 5, 4), 
      (0, "male", 37, 15, "yes", 4, 20, 5, 4), 
      (0, "female", 37, 15, "yes", 2, 14, 1, 3), 
      (0, "male", 22, 4, "yes", 1, 18, 5, 4), 
      (0, "male", 37, 15, "yes", 4, 17, 5, 3), 
      (0, "female", 22, 1.5, "no", 2, 14, 4, 5), 
      (0, "male", 52, 15, "yes", 4, 14, 6, 2), 
      (0, "female", 22, 1.5, "no", 4, 17, 5, 5), 
      (0, "male", 32, 4, "yes", 5, 14, 3, 5), 
      (0, "male", 32, 4, "yes", 2, 14, 3, 5), 
      (0, "female", 22, 1.5, "no", 3, 16, 6, 5), 
      (0, "male", 27, 0.75, "no", 2, 18, 3, 3), 
      (0, "female", 22, 7, "yes", 2, 14, 5, 2), 
      (0, "female", 27, 0.75, "no", 2, 17, 5, 3), 
      (0, "female", 37, 15, "yes", 4, 12, 1, 2), 
      (0, "female", 22, 1.5, "no", 1, 14, 1, 5), 
      (0, "female", 37, 10, "no", 2, 12, 4, 4), 
      (0, "female", 37, 15, "yes", 4, 18, 5, 3), 
      (0, "female", 42, 15, "yes", 3, 12, 3, 3), 
      (0, "male", 22, 4, "no", 2, 18, 5, 5), 
      (0, "male", 52, 7, "yes", 2, 20, 6, 2), 
      (0, "male", 27, 0.75, "no", 2, 17, 5, 5), 
      (0, "female", 27, 4, "no", 2, 17, 4, 5), 
      (0, "male", 42, 1.5, "no", 5, 20, 6, 5), 
      (0, "male", 22, 1.5, "no", 4, 17, 6, 5), 
      (0, "male", 22, 4, "no", 4, 17, 5, 3), 
      (0, "female", 22, 4, "yes", 1, 14, 5, 4), 
      (0, "male", 37, 15, "yes", 5, 20, 4, 5), 
      (0, "female", 37, 10, "yes", 3, 16, 6, 3), 
      (0, "male", 42, 15, "yes", 4, 17, 6, 5), 
      (0, "female", 47, 15, "yes", 4, 17, 5, 5), 
      (0, "male", 22, 1.5, "no", 4, 16, 5, 4), 
      (0, "female", 32, 10, "yes", 3, 12, 1, 4), 
      (0, "female", 22, 7, "yes", 1, 14, 3, 5), 
      (0, "female", 32, 10, "yes", 4, 17, 5, 4), 
      (0, "male", 27, 1.5, "yes", 2, 16, 2, 4), 
      (0, "male", 37, 15, "yes", 4, 14, 5, 5), 
      (0, "male", 42, 4, "yes", 3, 14, 4, 5), 
      (0, "female", 37, 15, "yes", 5, 14, 5, 4), 
      (0, "female", 32, 7, "yes", 4, 17, 5, 5), 
      (0, "female", 42, 15, "yes", 4, 18, 6, 5), 
      (0, "male", 27, 4, "no", 4, 18, 6, 4), 
      (0, "male", 22, 0.75, "no", 4, 18, 6, 5), 
      (0, "male", 27, 4, "yes", 4, 14, 5, 3), 
      (0, "female", 22, 0.75, "no", 5, 18, 1, 5), 
      (0, "female", 52, 15, "yes", 5, 9, 5, 5), 
      (0, "male", 32, 10, "yes", 3, 14, 5, 5), 
      (0, "female", 37, 15, "yes", 4, 16, 4, 4), 
      (0, "male", 32, 7, "yes", 2, 20, 5, 4), 
      (0, "female", 42, 15, "yes", 3, 18, 1, 4), 
      (0, "male", 32, 15, "yes", 1, 16, 5, 5), 
      (0, "male", 27, 4, "yes", 3, 18, 5, 5), 
      (0, "female", 32, 15, "yes", 4, 12, 3, 4), 
      (0, "male", 22, 0.75, "yes", 3, 14, 2, 4), 
      (0, "female", 22, 1.5, "no", 3, 16, 5, 3), 
      (0, "female", 42, 15, "yes", 4, 14, 3, 5), 
      (0, "female", 52, 15, "yes", 3, 16, 5, 4), 
      (0, "male", 37, 15, "yes", 5, 20, 6, 4), 
      (0, "female", 47, 15, "yes", 4, 12, 2, 3), 
      (0, "male", 57, 15, "yes", 2, 20, 6, 4), 
      (0, "male", 32, 7, "yes", 4, 17, 5, 5), 
      (0, "female", 27, 7, "yes", 4, 17, 1, 4), 
      (0, "male", 22, 1.5, "no", 1, 18, 6, 5), 
      (0, "female", 22, 4, "yes", 3, 9, 1, 4), 
      (0, "female", 22, 1.5, "no", 2, 14, 1, 5), 
      (0, "male", 42, 15, "yes", 2, 20, 6, 4), 
      (0, "male", 57, 15, "yes", 4, 9, 2, 4), 
      (0, "female", 27, 7, "yes", 2, 18, 1, 5), 
      (0, "female", 22, 4, "yes", 3, 14, 1, 5), 
      (0, "male", 37, 15, "yes", 4, 14, 5, 3), 
      (0, "male", 32, 7, "yes", 1, 18, 6, 4), 
      (0, "female", 22, 1.5, "no", 2, 14, 5, 5), 
      (0, "female", 22, 1.5, "yes", 3, 12, 1, 3), 
      (0, "male", 52, 15, "yes", 2, 14, 5, 5), 
      (0, "female", 37, 15, "yes", 2, 14, 1, 1), 
      (0, "female", 32, 10, "yes", 2, 14, 5, 5), 
      (0, "male", 42, 15, "yes", 4, 20, 4, 5), 
      (0, "female", 27, 4, "yes", 3, 18, 4, 5), 
      (0, "male", 37, 15, "yes", 4, 20, 6, 5), 
      (0, "male", 27, 1.5, "no", 3, 18, 5, 5), 
      (0, "female", 22, 0.125, "no", 2, 16, 6, 3), 
      (0, "male", 32, 10, "yes", 2, 20, 6, 3), 
      (0, "female", 27, 4, "no", 4, 18, 5, 4), 
      (0, "female", 27, 7, "yes", 2, 12, 5, 1), 
      (0, "male", 32, 4, "yes", 5, 18, 6, 3), 
      (0, "female", 37, 15, "yes", 2, 17, 5, 5), 
      (0, "male", 47, 15, "no", 4, 20, 6, 4), 
      (0, "male", 27, 1.5, "no", 1, 18, 5, 5), 
      (0, "male", 37, 15, "yes", 4, 20, 6, 4), 
      (0, "female", 32, 15, "yes", 4, 18, 1, 4), 
      (0, "female", 32, 7, "yes", 4, 17, 5, 4), 
      (0, "female", 42, 15, "yes", 3, 14, 1, 3), 
      (0, "female", 27, 7, "yes", 3, 16, 1, 4), 
      (0, "male", 27, 1.5, "no", 3, 16, 4, 2), 
      (0, "male", 22, 1.5, "no", 3, 16, 3, 5), 
      (0, "male", 27, 4, "yes", 3, 16, 4, 2), 
      (0, "female", 27, 7, "yes", 3, 12, 1, 2), 
      (0, "female", 37, 15, "yes", 2, 18, 5, 4), 
      (0, "female", 37, 7, "yes", 3, 14, 4, 4), 
      (0, "male", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "male", 37, 15, "yes", 5, 20, 5, 4), 
      (0, "female", 22, 1.5, "no", 4, 16, 5, 3), 
      (0, "female", 32, 10, "yes", 4, 16, 1, 5), 
      (0, "male", 27, 4, "no", 2, 17, 5, 3), 
      (0, "female", 22, 0.417, "no", 4, 14, 5, 5), 
      (0, "female", 27, 4, "no", 2, 18, 5, 5), 
      (0, "male", 37, 15, "yes", 4, 18, 5, 3), 
      (0, "male", 37, 10, "yes", 5, 20, 7, 4), 
      (0, "female", 27, 7, "yes", 2, 14, 4, 2), 
      (0, "male", 32, 4, "yes", 2, 16, 5, 5), 
      (0, "male", 32, 4, "yes", 2, 16, 6, 4), 
      (0, "male", 22, 1.5, "no", 3, 18, 4, 5), 
      (0, "female", 22, 4, "yes", 4, 14, 3, 4), 
      (0, "female", 17.5, 0.75, "no", 2, 18, 5, 4), 
      (0, "male", 32, 10, "yes", 4, 20, 4, 5), 
      (0, "female", 32, 0.75, "no", 5, 14, 3, 3), 
      (0, "male", 37, 15, "yes", 4, 17, 5, 3), 
      (0, "male", 32, 4, "no", 3, 14, 4, 5), 
      (0, "female", 27, 1.5, "no", 2, 17, 3, 2), 
      (0, "female", 22, 7, "yes", 4, 14, 1, 5), 
      (0, "male", 47, 15, "yes", 5, 14, 6, 5), 
      (0, "male", 27, 4, "yes", 1, 16, 4, 4), 
      (0, "female", 37, 15, "yes", 5, 14, 1, 3), 
      (0, "male", 42, 4, "yes", 4, 18, 5, 5), 
      (0, "female", 32, 4, "yes", 2, 14, 1, 5), 
      (0, "male", 52, 15, "yes", 2, 14, 7, 4), 
      (0, "female", 22, 1.5, "no", 2, 16, 1, 4), 
      (0, "male", 52, 15, "yes", 4, 12, 2, 4), 
      (0, "female", 22, 0.417, "no", 3, 17, 1, 5), 
      (0, "female", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "male", 27, 4, "yes", 4, 20, 6, 4), 
      (0, "female", 32, 15, "yes", 4, 14, 1, 5), 
      (0, "female", 27, 1.5, "no", 2, 16, 3, 5), 
      (0, "male", 32, 4, "no", 1, 20, 6, 5), 
      (0, "male", 37, 15, "yes", 3, 20, 6, 4), 
      (0, "female", 32, 10, "no", 2, 16, 6, 5), 
      (0, "female", 32, 10, "yes", 5, 14, 5, 5), 
      (0, "male", 37, 1.5, "yes", 4, 18, 5, 3), 
      (0, "male", 32, 1.5, "no", 2, 18, 4, 4), 
      (0, "female", 32, 10, "yes", 4, 14, 1, 4), 
      (0, "female", 47, 15, "yes", 4, 18, 5, 4), 
      (0, "female", 27, 10, "yes", 5, 12, 1, 5), 
      (0, "male", 27, 4, "yes", 3, 16, 4, 5), 
      (0, "female", 37, 15, "yes", 4, 12, 4, 2), 
      (0, "female", 27, 0.75, "no", 4, 16, 5, 5), 
      (0, "female", 37, 15, "yes", 4, 16, 1, 5), 
      (0, "female", 32, 15, "yes", 3, 16, 1, 5), 
      (0, "female", 27, 10, "yes", 2, 16, 1, 5), 
      (0, "male", 27, 7, "no", 2, 20, 6, 5), 
      (0, "female", 37, 15, "yes", 2, 14, 1, 3), 
      (0, "male", 27, 1.5, "yes", 2, 17, 4, 4), 
      (0, "female", 22, 0.75, "yes", 2, 14, 1, 5), 
      (0, "male", 22, 4, "yes", 4, 14, 2, 4), 
      (0, "male", 42, 0.125, "no", 4, 17, 6, 4), 
      (0, "male", 27, 1.5, "yes", 4, 18, 6, 5), 
      (0, "male", 27, 7, "yes", 3, 16, 6, 3), 
      (0, "female", 52, 15, "yes", 4, 14, 1, 3), 
      (0, "male", 27, 1.5, "no", 5, 20, 5, 2), 
      (0, "female", 27, 1.5, "no", 2, 16, 5, 5), 
      (0, "female", 27, 1.5, "no", 3, 17, 5, 5), 
      (0, "male", 22, 0.125, "no", 5, 16, 4, 4), 
      (0, "female", 27, 4, "yes", 4, 16, 1, 5), 
      (0, "female", 27, 4, "yes", 4, 12, 1, 5), 
      (0, "female", 47, 15, "yes", 2, 14, 5, 5), 
      (0, "female", 32, 15, "yes", 3, 14, 5, 3), 
      (0, "male", 42, 7, "yes", 2, 16, 5, 5), 
      (0, "male", 22, 0.75, "no", 4, 16, 6, 4), 
      (0, "male", 27, 0.125, "no", 3, 20, 6, 5), 
      (0, "male", 32, 10, "yes", 3, 20, 6, 5), 
      (0, "female", 22, 0.417, "no", 5, 14, 4, 5), 
      (0, "female", 47, 15, "yes", 5, 14, 1, 4), 
      (0, "female", 32, 10, "yes", 3, 14, 1, 5), 
      (0, "male", 57, 15, "yes", 4, 17, 5, 5), 
      (0, "male", 27, 4, "yes", 3, 20, 6, 5), 
      (0, "female", 32, 7, "yes", 4, 17, 1, 5), 
      (0, "female", 37, 10, "yes", 4, 16, 1, 5), 
      (0, "female", 32, 10, "yes", 1, 18, 1, 4), 
      (0, "female", 22, 4, "no", 3, 14, 1, 4), 
      (0, "female", 27, 7, "yes", 4, 14, 3, 2), 
      (0, "male", 57, 15, "yes", 5, 18, 5, 2), 
      (0, "male", 32, 7, "yes", 2, 18, 5, 5), 
      (0, "female", 27, 1.5, "no", 4, 17, 1, 3), 
      (0, "male", 22, 1.5, "no", 4, 14, 5, 5), 
      (0, "female", 22, 1.5, "yes", 4, 14, 5, 4), 
      (0, "female", 32, 7, "yes", 3, 16, 1, 5), 
      (0, "female", 47, 15, "yes", 3, 16, 5, 4), 
      (0, "female", 22, 0.75, "no", 3, 16, 1, 5), 
      (0, "female", 22, 1.5, "yes", 2, 14, 5, 5), 
      (0, "female", 27, 4, "yes", 1, 16, 5, 5), 
      (0, "male", 52, 15, "yes", 4, 16, 5, 5), 
      (0, "male", 32, 10, "yes", 4, 20, 6, 5), 
      (0, "male", 47, 15, "yes", 4, 16, 6, 4), 
      (0, "female", 27, 7, "yes", 2, 14, 1, 2), 
      (0, "female", 22, 1.5, "no", 4, 14, 4, 5), 
      (0, "female", 32, 10, "yes", 2, 16, 5, 4), 
      (0, "female", 22, 0.75, "no", 2, 16, 5, 4), 
      (0, "female", 22, 1.5, "no", 2, 16, 5, 5), 
      (0, "female", 42, 15, "yes", 3, 18, 6, 4), 
      (0, "female", 27, 7, "yes", 5, 14, 4, 5), 
      (0, "male", 42, 15, "yes", 4, 16, 4, 4), 
      (0, "female", 57, 15, "yes", 3, 18, 5, 2), 
      (0, "male", 42, 15, "yes", 3, 18, 6, 2), 
      (0, "female", 32, 7, "yes", 2, 14, 1, 2), 
      (0, "male", 22, 4, "no", 5, 12, 4, 5), 
      (0, "female", 22, 1.5, "no", 1, 16, 6, 5), 
      (0, "female", 22, 0.75, "no", 1, 14, 4, 5), 
      (0, "female", 32, 15, "yes", 4, 12, 1, 5), 
      (0, "male", 22, 1.5, "no", 2, 18, 5, 3), 
      (0, "male", 27, 4, "yes", 5, 17, 2, 5), 
      (0, "female", 27, 4, "yes", 4, 12, 1, 5), 
      (0, "male", 42, 15, "yes", 5, 18, 5, 4), 
      (0, "male", 32, 1.5, "no", 2, 20, 7, 3), 
      (0, "male", 57, 15, "no", 4, 9, 3, 1), 
      (0, "male", 37, 7, "no", 4, 18, 5, 5), 
      (0, "male", 52, 15, "yes", 2, 17, 5, 4), 
      (0, "male", 47, 15, "yes", 4, 17, 6, 5), 
      (0, "female", 27, 7, "no", 2, 17, 5, 4), 
      (0, "female", 27, 7, "yes", 4, 14, 5, 5), 
      (0, "female", 22, 4, "no", 2, 14, 3, 3), 
      (0, "male", 37, 7, "yes", 2, 20, 6, 5), 
      (0, "male", 27, 7, "no", 4, 12, 4, 3), 
      (0, "male", 42, 10, "yes", 4, 18, 6, 4), 
      (0, "female", 22, 1.5, "no", 3, 14, 1, 5), 
      (0, "female", 22, 4, "yes", 2, 14, 1, 3), 
      (0, "female", 57, 15, "no", 4, 20, 6, 5), 
      (0, "male", 37, 15, "yes", 4, 14, 4, 3), 
      (0, "female", 27, 7, "yes", 3, 18, 5, 5), 
      (0, "female", 17.5, 10, "no", 4, 14, 4, 5), 
      (0, "male", 22, 4, "yes", 4, 16, 5, 5), 
      (0, "female", 27, 4, "yes", 2, 16, 1, 4), 
      (0, "female", 37, 15, "yes", 2, 14, 5, 1), 
      (0, "female", 22, 1.5, "no", 5, 14, 1, 4), 
      (0, "male", 27, 7, "yes", 2, 20, 5, 4), 
      (0, "male", 27, 4, "yes", 4, 14, 5, 5), 
      (0, "male", 22, 0.125, "no", 1, 16, 3, 5), 
      (0, "female", 27, 7, "yes", 4, 14, 1, 4), 
      (0, "female", 32, 15, "yes", 5, 16, 5, 3), 
      (0, "male", 32, 10, "yes", 4, 18, 5, 4), 
      (0, "female", 32, 15, "yes", 2, 14, 3, 4), 
      (0, "female", 22, 1.5, "no", 3, 17, 5, 5), 
      (0, "male", 27, 4, "yes", 4, 17, 4, 4), 
      (0, "female", 52, 15, "yes", 5, 14, 1, 5), 
      (0, "female", 27, 7, "yes", 2, 12, 1, 2), 
      (0, "female", 27, 7, "yes", 3, 12, 1, 4), 
      (0, "female", 42, 15, "yes", 2, 14, 1, 4), 
      (0, "female", 42, 15, "yes", 4, 14, 5, 4), 
      (0, "male", 27, 7, "yes", 4, 14, 3, 3), 
      (0, "male", 27, 7, "yes", 2, 20, 6, 2), 
      (0, "female", 42, 15, "yes", 3, 12, 3, 3), 
      (0, "male", 27, 4, "yes", 3, 16, 3, 5), 
      (0, "female", 27, 7, "yes", 3, 14, 1, 4), 
      (0, "female", 22, 1.5, "no", 2, 14, 4, 5), 
      (0, "female", 27, 4, "yes", 4, 14, 1, 4), 
      (0, "female", 22, 4, "no", 4, 14, 5, 5), 
      (0, "female", 22, 1.5, "no", 2, 16, 4, 5), 
      (0, "male", 47, 15, "no", 4, 14, 5, 4), 
      (0, "male", 37, 10, "yes", 2, 18, 6, 2), 
      (0, "male", 37, 15, "yes", 3, 17, 5, 4), 
      (0, "female", 27, 4, "yes", 2, 16, 1, 4), 
      (3, "male", 27, 1.5, "no", 3, 18, 4, 4), 
      (3, "female", 27, 4, "yes", 3, 17, 1, 5), 
      (7, "male", 37, 15, "yes", 5, 18, 6, 2), 
      (12, "female", 32, 10, "yes", 3, 17, 5, 2), 
      (1, "male", 22, 0.125, "no", 4, 16, 5, 5), 
      (1, "female", 22, 1.5, "yes", 2, 14, 1, 5), 
      (12, "male", 37, 15, "yes", 4, 14, 5, 2), 
      (7, "female", 22, 1.5, "no", 2, 14, 3, 4), 
      (2, "male", 37, 15, "yes", 2, 18, 6, 4), 
      (3, "female", 32, 15, "yes", 4, 12, 3, 2), 
      (1, "female", 37, 15, "yes", 4, 14, 4, 2), 
      (7, "female", 42, 15, "yes", 3, 17, 1, 4), 
      (12, "female", 42, 15, "yes", 5, 9, 4, 1), 
      (12, "male", 37, 10, "yes", 2, 20, 6, 2), 
      (12, "female", 32, 15, "yes", 3, 14, 1, 2), 
      (3, "male", 27, 4, "no", 1, 18, 6, 5), 
      (7, "male", 37, 10, "yes", 2, 18, 7, 3), 
      (7, "female", 27, 4, "no", 3, 17, 5, 5), 
      (1, "male", 42, 15, "yes", 4, 16, 5, 5), 
      (1, "female", 47, 15, "yes", 5, 14, 4, 5), 
      (7, "female", 27, 4, "yes", 3, 18, 5, 4), 
      (1, "female", 27, 7, "yes", 5, 14, 1, 4), 
      (12, "male", 27, 1.5, "yes", 3, 17, 5, 4), 
      (12, "female", 27, 7, "yes", 4, 14, 6, 2), 
      (3, "female", 42, 15, "yes", 4, 16, 5, 4), 
      (7, "female", 27, 10, "yes", 4, 12, 7, 3), 
      (1, "male", 27, 1.5, "no", 2, 18, 5, 2), 
      (1, "male", 32, 4, "no", 4, 20, 6, 4), 
      (1, "female", 27, 7, "yes", 3, 14, 1, 3), 
      (3, "female", 32, 10, "yes", 4, 14, 1, 4), 
      (3, "male", 27, 4, "yes", 2, 18, 7, 2), 
      (1, "female", 17.5, 0.75, "no", 5, 14, 4, 5), 
      (1, "female", 32, 10, "yes", 4, 18, 1, 5), 
      (7, "female", 32, 7, "yes", 2, 17, 6, 4), 
      (7, "male", 37, 15, "yes", 2, 20, 6, 4), 
      (7, "female", 37, 10, "no", 1, 20, 5, 3), 
      (12, "female", 32, 10, "yes", 2, 16, 5, 5), 
      (7, "male", 52, 15, "yes", 2, 20, 6, 4), 
      (7, "female", 42, 15, "yes", 1, 12, 1, 3), 
      (1, "male", 52, 15, "yes", 2, 20, 6, 3), 
      (2, "male", 37, 15, "yes", 3, 18, 6, 5), 
      (12, "female", 22, 4, "no", 3, 12, 3, 4), 
      (12, "male", 27, 7, "yes", 1, 18, 6, 2), 
      (1, "male", 27, 4, "yes", 3, 18, 5, 5), 
      (12, "male", 47, 15, "yes", 4, 17, 6, 5), 
      (12, "female", 42, 15, "yes", 4, 12, 1, 1), 
      (7, "male", 27, 4, "no", 3, 14, 3, 4), 
      (7, "female", 32, 7, "yes", 4, 18, 4, 5), 
      (1, "male", 32, 0.417, "yes", 3, 12, 3, 4), 
      (3, "male", 47, 15, "yes", 5, 16, 5, 4), 
      (12, "male", 37, 15, "yes", 2, 20, 5, 4), 
      (7, "male", 22, 4, "yes", 2, 17, 6, 4), 
      (1, "male", 27, 4, "no", 2, 14, 4, 5), 
      (7, "female", 52, 15, "yes", 5, 16, 1, 3), 
      (1, "male", 27, 4, "no", 3, 14, 3, 3), 
      (1, "female", 27, 10, "yes", 4, 16, 1, 4), 
      (1, "male", 32, 7, "yes", 3, 14, 7, 4), 
      (7, "male", 32, 7, "yes", 2, 18, 4, 1), 
      (3, "male", 22, 1.5, "no", 1, 14, 3, 2), 
      (7, "male", 22, 4, "yes", 3, 18, 6, 4), 
      (7, "male", 42, 15, "yes", 4, 20, 6, 4), 
      (2, "female", 57, 15, "yes", 1, 18, 5, 4), 
      (7, "female", 32, 4, "yes", 3, 18, 5, 2), 
      (1, "male", 27, 4, "yes", 1, 16, 4, 4), 
      (7, "male", 32, 7, "yes", 4, 16, 1, 4), 
      (2, "male", 57, 15, "yes", 1, 17, 4, 4), 
      (7, "female", 42, 15, "yes", 4, 14, 5, 2), 
      (7, "male", 37, 10, "yes", 1, 18, 5, 3), 
      (3, "male", 42, 15, "yes", 3, 17, 6, 1), 
      (1, "female", 52, 15, "yes", 3, 14, 4, 4), 
      (2, "female", 27, 7, "yes", 3, 17, 5, 3), 
      (12, "male", 32, 7, "yes", 2, 12, 4, 2), 
      (1, "male", 22, 4, "no", 4, 14, 2, 5), 
      (3, "male", 27, 7, "yes", 3, 18, 6, 4), 
      (12, "female", 37, 15, "yes", 1, 18, 5, 5), 
      (7, "female", 32, 15, "yes", 3, 17, 1, 3), 
      (7, "female", 27, 7, "no", 2, 17, 5, 5), 
      (1, "female", 32, 7, "yes", 3, 17, 5, 3), 
      (1, "male", 32, 1.5, "yes", 2, 14, 2, 4), 
      (12, "female", 42, 15, "yes", 4, 14, 1, 2), 
      (7, "male", 32, 10, "yes", 3, 14, 5, 4), 
      (7, "male", 37, 4, "yes", 1, 20, 6, 3), 
      (1, "female", 27, 4, "yes", 2, 16, 5, 3), 
      (12, "female", 42, 15, "yes", 3, 14, 4, 3), 
      (1, "male", 27, 10, "yes", 5, 20, 6, 5), 
      (12, "male", 37, 10, "yes", 2, 20, 6, 2), 
      (12, "female", 27, 7, "yes", 1, 14, 3, 3), 
      (3, "female", 27, 7, "yes", 4, 12, 1, 2), 
      (3, "male", 32, 10, "yes", 2, 14, 4, 4), 
      (12, "female", 17.5, 0.75, "yes", 2, 12, 1, 3), 
      (12, "female", 32, 15, "yes", 3, 18, 5, 4), 
      (2, "female", 22, 7, "no", 4, 14, 4, 3), 
      (1, "male", 32, 7, "yes", 4, 20, 6, 5), 
      (7, "male", 27, 4, "yes", 2, 18, 6, 2), 
      (1, "female", 22, 1.5, "yes", 5, 14, 5, 3), 
      (12, "female", 32, 15, "no", 3, 17, 5, 1), 
      (12, "female", 42, 15, "yes", 2, 12, 1, 2), 
      (7, "male", 42, 15, "yes", 3, 20, 5, 4), 
      (12, "male", 32, 10, "no", 2, 18, 4, 2), 
      (12, "female", 32, 15, "yes", 3, 9, 1, 1), 
      (7, "male", 57, 15, "yes", 5, 20, 4, 5), 
      (12, "male", 47, 15, "yes", 4, 20, 6, 4), 
      (2, "female", 42, 15, "yes", 2, 17, 6, 3), 
      (12, "male", 37, 15, "yes", 3, 17, 6, 3), 
      (12, "male", 37, 15, "yes", 5, 17, 5, 2), 
      (7, "male", 27, 10, "yes", 2, 20, 6, 4), 
      (2, "male", 37, 15, "yes", 2, 16, 5, 4), 
      (12, "female", 32, 15, "yes", 1, 14, 5, 2), 
      (7, "male", 32, 10, "yes", 3, 17, 6, 3), 
      (2, "male", 37, 15, "yes", 4, 18, 5, 1), 
      (7, "female", 27, 1.5, "no", 2, 17, 5, 5), 
      (3, "female", 47, 15, "yes", 2, 17, 5, 2), 
      (12, "male", 37, 15, "yes", 2, 17, 5, 4), 
      (12, "female", 27, 4, "no", 2, 14, 5, 5), 
      (2, "female", 27, 10, "yes", 4, 14, 1, 5), 
      (1, "female", 22, 4, "yes", 3, 16, 1, 3), 
      (12, "male", 52, 7, "no", 4, 16, 5, 5), 
      (2, "female", 27, 4, "yes", 1, 16, 3, 5), 
      (7, "female", 37, 15, "yes", 2, 17, 6, 4), 
      (2, "female", 27, 4, "no", 1, 17, 3, 1), 
      (12, "female", 17.5, 0.75, "yes", 2, 12, 3, 5), 
      (7, "female", 32, 15, "yes", 5, 18, 5, 4), 
      (7, "female", 22, 4, "no", 1, 16, 3, 5), 
      (2, "male", 32, 4, "yes", 4, 18, 6, 4), 
      (1, "female", 22, 1.5, "yes", 3, 18, 5, 2), 
      (3, "female", 42, 15, "yes", 2, 17, 5, 4), 
      (1, "male", 32, 7, "yes", 4, 16, 4, 4), 
      (12, "male", 37, 15, "no", 3, 14, 6, 2), 
      (1, "male", 42, 15, "yes", 3, 16, 6, 3), 
      (1, "male", 27, 4, "yes", 1, 18, 5, 4), 
      (2, "male", 37, 15, "yes", 4, 20, 7, 3), 
      (7, "male", 37, 15, "yes", 3, 20, 6, 4), 
      (3, "male", 22, 1.5, "no", 2, 12, 3, 3), 
      (3, "male", 32, 4, "yes", 3, 20, 6, 2), 
      (2, "male", 32, 15, "yes", 5, 20, 6, 5), 
      (12, "female", 52, 15, "yes", 1, 18, 5, 5), 
      (12, "male", 47, 15, "no", 1, 18, 6, 5), 
      (3, "female", 32, 15, "yes", 4, 16, 4, 4), 
      (7, "female", 32, 15, "yes", 3, 14, 3, 2), 
      (7, "female", 27, 7, "yes", 4, 16, 1, 2), 
      (12, "male", 42, 15, "yes", 3, 18, 6, 2), 
      (7, "female", 42, 15, "yes", 2, 14, 3, 2), 
      (12, "male", 27, 7, "yes", 2, 17, 5, 4), 
      (3, "male", 32, 10, "yes", 4, 14, 4, 3), 
      (7, "male", 47, 15, "yes", 3, 16, 4, 2), 
      (1, "male", 22, 1.5, "yes", 1, 12, 2, 5), 
      (7, "female", 32, 10, "yes", 2, 18, 5, 4), 
      (2, "male", 32, 10, "yes", 2, 17, 6, 5), 
      (2, "male", 22, 7, "yes", 3, 18, 6, 2), 
      (1, "female", 32, 15, "yes", 3, 14, 1, 5))
       
val data = dataList.toDF("affairs", "gender", "age", "yearsmarried", "children", "religiousness", "education", "occupation", "rating")

　　GBT建模

data.createOrReplaceTempView("data")
 
// 字符类型转换成数值  
val labelWhere = "affairs as label"
val genderWhere = "case when gender='female' then 0 else cast(1 as double) end as gender"
val childrenWhere = "case when children='no' then 0 else cast(1 as double) end as children"
 
val dataLabelDF = spark.sql(s"select $labelWhere, $genderWhere,age,yearsmarried,$childrenWhere,religiousness,education,occupation,rating from data")
 
val featuresArray = Array("gender", "age", "yearsmarried", "children", "religiousness", "education", "occupation", "rating")
 
// 字段转换成特征向量  
val assembler = new VectorAssembler().setInputCols(featuresArray).setOutputCol("features")
val vecDF: DataFrame = assembler.transform(dataLabelDF)
vecDF.show(10, truncate = false)
 
// 将数据分为训练和测试集（30%进行测试） 
val Array(trainingDF, testDF) = vecDF.randomSplit(Array(0.7, 0.3))
 
// 自动识别分类的特征，并对它们进行索引  
// 具有大于5个不同的值的特征被视为连续。  
val featureIndexer = new VectorIndexer().setInputCol("features").setOutputCol("indexedFeatures").setMaxCategories(5)
 
 
 
// 训练GBT模型
val gbt = new GBTRegressor().setLabelCol("label").setFeaturesCol("indexedFeatures").setImpurity("variance").setLossType("squared").setMaxIter(100).setMinInstancesPerNode(100)
 
 
 
// Chain indexer and GBT in a Pipeline.
val pipeline = new Pipeline().setStages(Array(featureIndexer, gbt))
 
// Train model. This also runs the indexer.
val model = pipeline.fit(trainingDF)
 
// 做出预测
val predictions = model.transform(testDF)
 
// 预测样本展示
predictions.select("prediction", "label", "features").show(20,false)
 
// 选择（预测标签，实际标签），并计算测试误差。
val evaluator = new RegressionEvaluator().setLabelCol("label").setPredictionCol("prediction").setMetricName("rmse")
val rmse = evaluator.evaluate(predictions)
println("Root Mean Squared Error (RMSE) on test data = " + rmse)
 
val gbtModel = model.stages(1).asInstanceOf[GBTRegressionModel]
println("Learned regression GBT model:\n" + gbtModel.toDebugString)

　　

转载于:https://www.cnblogs.com/xiaoma0529/p/7230297.html

机器学习(Machine Learning)与深度学习(Deep Learning)资料汇总
ZhangPY的专栏
02-28 7302
《Brief History of Machine Learning》 介绍:这是一篇介绍机器学习历史的文章，介绍很全面，从感知机、神经网络、决策树、SVM、Adaboost到随机森林、Deep Learning. 《Deep Learning in Neural Networks: An Overview》 介绍:这是瑞士人工智能实验室Jurgen Schmidhuber写的最
GEE：梯度提升树（Gradient Boosting Tree）回归教程（样本点、特征添加、训练、精度、参数优化）
养乐多的博客
02-13 982
本文将介绍在Google Earth Engine （GEE）平台上进行梯度提升树（Gradient Boosting Tree）回归的方法和代码，其中包括样本点格式介绍，加入特征变量(各种指数、纹理特征、时间序列特征、物候特征等)，训练和应用回归模型，优化回归算法的参数（绘制最优参数分布图）等步骤的方法和代码。
spark.mllib源码阅读-分类算法5-GradientBoostedTrees
Forever-守望
04-19 4549
Gradient-Boosted Trees(GBT或者GBDT) 和 RandomForests 都属于集成学习的范畴，相比于单个模型有限的表达能力，组合多个base model后表达能力更加丰富。关于集成学习的理论知识，包括GBT和Random Forests的一些比较好的参考资料：周志华教授的"Ensemble Methods: Foundations and Algorithms",系统的
Gradient Boosted Decision Trees（GBDT）详解
Learning from the mistakes
09-16 1万+
感受 GBDT集成方法的一种，就是根据每次剩余的残差，即损失函数的值。在残差减少的方向上建立一个新的模型的方法，直到达到一定拟合精度后停止。我找了一个相关的例子来帮助理解。本文结合了多篇博客和书，试图完整介绍GBDT的内容，欢迎大家来指正。 介绍 GBDT是一个应用很广泛的算法，可以用来做分类、回归。GBDT这个算法还有其它名字，如MART(Multiple Additiv...
Gradient Boosted Decision Tree (GBDT).
qq_25678319的博客
11-25 211
理论： https://www.cnblogs.com/peizhe123/p/5086128.html  参数： http://www.cnblogs.com/joneswood/archive/2012/03/04/2379615.html
深入机器学习系列之：Gradient-boosted tree
数据猿
12-17 2186
导读梯度提升属于Boost算法的一种，也可以说是Boost算法的一种改进，它与传统的Boost有着很大的区别，它的每一次计算都是为了减少上一次的残差，而为了减少这些残差，...
【Spark】分类和回归算法-分类
buracag_mc的博客
09-03 1337
同步于Buracag的博客 本节主要讲Spark ML中关于分类算法的实现。示例的算法Demo包含：LR、DT、RF、GBTs、多层感知器、线性支持向量机、One-vs-Rest分类器以及NB等。 文章目录1. Logistic regression1.1 二分类LR1.2 多分类LR2. 决策树分类器3. 随机森林分类器4. 梯度提升树分类器5. 多层感知器分类器6. 线性支持向量机7. One...
5. PySpark分类模型简明教程
chendengyi2的博客
07-31 1751
Spark集群是一种分布式计算框架，它基于内存计算，提供了高效的数据抽象和并行计算能力，能够处理大规模数据集的批处理和实时处理任务。Spark采用内存存储中间计算结果，可减少迭代运算的磁盘I/O，并通过并行计算有向无环图的优化，使其运行速度比MapReduce快100倍；Spark可以使用Hadoop YARN和Apache Mesos作为其资源管理和调度器，可以从多种数据源读取数据，如HDFS、HBase、MySQL等。
机器学习 深度学习资料汇总（含文档，数据集，代码等） 三
qq_44906416的博客
04-21 898
机器学习 深度学习资料汇总（含文档，数据集，代码等） 三
机器学习(Machine Learning)&深度学习(Deep Learning)资料
热门推荐
weixin_30756499的博客
07-07 1万+
##机器学习(Machine Learning)&深度学习(Deep Learning)资料(Chapter 1)---#####注:机器学习资料[篇目一](https://github.com/ty4z2008/Qix/blob/master/dl.md)共500条,[篇目二](https://github.com/ty4z2008/Qix/blob/master/dl2.md)开始更新...
Python-GradientBoostedTrees的一个简单的纯Python实现
08-11
TinyGBT（Tiny Gradient Boosted Trees）是用200行纯Python代码编写的梯度提升树实现。
gradient_boosted_decision_tree
qq_30534935的博客
04-23 306
来源：https://github.com/aymericdamien/TensorFlow-Examples """ Gradient Boosted Decision Tree (GBDT). Implement a Gradient Boosted Decision tree with TensorFlow to classify handwritten digit images. Thi...
Opencv2.4.9源码分析——Gradient Boosted Trees
赵春江的专栏
05-20 9462
一、原理   梯度提升树（GBT，Gradient Boosted Trees，或称为梯度提升决策树）算法是由Friedman于1999年首次完整的提出，该算法可以实现回归、分类和排序。GBT的优点是特征属性无需进行归一化处理，预测速度快，可以应用不同的损失函数等。 从它的名字就可以看出，GBT包括三个机器学习的优化算法：决策树方法、提升方法和梯度下降法。前两种算法在我以前的文章中都有详细的
python 梯度提升树_梯度提升树（Gradient Boosted Trees）：模型理解
weixin_39795065的博客
12-09 996
Note:我们的 TensorFlow 社区翻译了这些文档。因为社区翻译是尽力而为， 所以无法保证它们是最准确的，并且反映了最新的官方英文文档。如果您有改进此翻译的建议， 请提交 pull request 到tensorflow/docs GitHub 仓库。要志愿地撰写或者审核译文，请加入docs-zh-cn@tensorflow.org Google Group。对于梯度提升模型(Gradie...
Gradient boosted decision tree
Du_Shuang的博客
10-11 359
1.Adboost decision tree Gradient boosted decision 就是将Adaboost和decision Tree结合起来的算法。 Adaboost是根据前一步求出的g的错误率来提高错误数据在下一笔训练数据里面的权重来不断训练处新g的。 但是decision tree没有权重的概念我们该怎么办呢？我们的方法是根据权重来对数据进行取样，比如错误率本来是10，那...
梯度提升树（Gradient Boosting Trees）详解
最新发布
不迁怒，不贰过。小知识，大智慧。
11-18 1483
在当今数据驱动的时代，机器学习已成为解决复杂问题的重要工具。随着数据量的激增和计算能力的提升，越来越多的算法被提出以应对各种挑战。其中，梯度提升树（Gradient Boosting Trees, GBT）因其卓越的性能和灵活性，迅速崛起为最受欢迎的算法之一。GBT通过将多个弱学习器（通常是决策树）结合在一起，逐步优化模型，能够有效地处理回归和分类任务。   本文旨在深入探讨梯度提升树的基本原理、数学模型、实现步骤及其应用场景。通过对这些关键知识点的详细分析，读者将能够更好地理解GBT的工作机制，并在实际
spark源码分析--GradientBoostedTrees和RandomForest
fengkuang的一亩三分地
07-24 922
spark源码分析–GradientBoostedTrees和RandomForest GradientBoostedTree是spark mllib中的一个树模型，源码见GradientBoostedTrees.scala。该模型仅适用于回归和二分类问题。 训练调用方法 import org.apache.spark.mllib.tree.GradientBoostedTrees /...
Gradient Boosted Trees 的视觉指南
woliuqiangdong的博客
01-19 242
　　使用MNIST数据库直观地解释GBT 　　大家好，欢迎回到"机器学习视觉指南"系列中的另一篇文章！我们将学习另一种流行的模型集成方法，称为梯度增强树。如果您还没有的话，请查看上一篇文章，以了解有关随机森林的知识，我们在其中介绍了模型集成和决策树的概念，决策树是这些模型的基础。观看下面的视频，获取本文的4分钟演练以及该系列以前的视频的链接！ 　　我们将使用梯度增强树进行分类：具体来说，是识别图像中绘制的数字。我们将使用MNIST，这是图像处理中常用的大型手写图像数据库。它包含60,000个训练图像和1
Spark MLlib中的随机森林与梯度提升树实战
在机器学习领域，随机森林和梯度提升树（Gradient Boosting Trees, GBTs）是两种广泛应用的集成学习方法。集成学习通过组合多个弱学习器，如决策树，来创建一个强大的强学习器。在Spark的MLlib库1.2版本中，这两种...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值