如何在Spark中使用gbdt模型分布式预测

1 训练gbdt模型

我们可以基于lightgbm快速的训练一个gbdt模型，训练相对比较简单，只要把训练样本处理好，几行代码可以快速训练好模型，如下是训练一个多分类模型训练核心代码如下：

import lightgbm as lgb
import joblib
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
#假设处理好的训练样本为train.csv
df = pd.read_csv('./train.csv')
X = pd.drop(['label'],axis=1)
Y = df.label
# split data for val
x_train,x_val,y_train,y_val = train_test_split(X,Y,test_size=0.2,random_state=123)
# train model
cate_features=['sex','brand']
train_data = train_data = lgb.Dataset(x_train,label=y_train,categoryical_featrues=cate_features)
params = {
   
	'objective':'multiclass',
	'learning_rate':0.1,
	'n_estimators':100,
	'num_class':23
	}
model = lgb.train(params, train_data,100)
#predict val
y_pred = model.predict(x_val)
y_pred = y_pred.argmax(axis=1)

# acc
acc = accuracy_score(y_val, y_pred)
print(acc)

# feature importance
feature_name = model.feature_name()
feature_importance = model.feature_importance()
feature_score = dict(zip(feature_name, feature_importance))
feature_score_sort = sorted(feature_score.items(),key=lambda