模型融合:对调参完成的模型进行融合,一般在整个过程的后期进行,基于多个比较好的模型,融合后才有好的效果。
方法:
-
简单加权融合:
回归(分类概率):
算术平均融合(Arithmetic mean),几何平均融合(Geometric mean);权重的差异不要太大分类模型:
投票(Voting),分为硬投票和软投票,原理是少数服从多数。硬投票:对多个模型直接进行投票,不区分模型结果的相对重要度,最终投票数最多的类为最终被预测的类。软投票:增加了设置权重的功能,可以为不同模型设置不同权重,进而区别模型不同的重要度综合:
排序融合(Rank averaging),log融合 -
Stacking/Blending:
构建多层模型,并利用预测结果再拟合预测。
先学习一个初级的模型,然后用初级的模型训练次级的模型,从而得到完整的stacking模型。
为了防止过拟合,通常次级的模型尽量选择简单的线性模型 -
boosting/bagging(在xgboost,Adaboost,GBDT中已经用到):
多树的提升方法
不同的方法需要多尝试才知道结果,有时简单融合不容易过拟合。
示例:
继续之前的(二)特征工程之后,
构建XGBoost、LightGBM、GBDT三个模型,
def build_model_gbdt(x_train,y_train):
estimator =GradientBoostingRegressor(loss='ls',subsample= 0.85,max_depth= 5,n_estimators = 100)
param_grid = {
'learning_rate': [0.05,0.08,0.1,0.2],
}
gbdt = GridSearchCV(estimator, param_grid,cv=3)
gbdt.fit(x_train,y_train)
print(gbdt.best_params_)
# print(gbdt.best_estimator_ )
return gbdt
def build_model_xgb(x_train,y_train):
model = xgb.XGBRegressor(n_estimators=120, learning_rate=0.08, gamma=0, subsample=0.8,\
colsample_bytree=0.9, max_depth=5) #, objective ='reg:squarederror'
model.fit(x_train, y_train)
return model
def build_model_lgb(x_train,y_train):
estimator = lgb.LGBMRegressor(num_leaves=63,n_estimators = 100)
param_grid = {
'learning_rate': [0.01, 0.05, 0.1],
}
gbm = GridSearchCV(estimator, param_grid)
gbm.fit(x_train, y_train)
return gbm
model_xgb = build_model_xgb(train_X,train_y)
model_lgb = build_model_lgb(train_X,train_y)
model_gbdt = build_model_gbdt(train_X,train_y)
分别用3个模型做预测
print('Predict xgb...')
train_xgb_pred = model_xgb.predict(train_X)
subA_xgb = model_xgb.predict(Test_data)
print('Predict lgb...')
train_lgb_pred = model_lgb.predict(train_X)
subA_lgb = model_lgb.predict(Test_data)
print('Predict GBDT...')
train_gbdt_pred = model_gbdt.predict(train_X)
subA_gbdt = model_gbdt.predict(Test_data)
用Stacking方法做模型融合
Strak_X_train = pd.DataFrame()
Strak_X_train['Method_1'] = train_lgb_pred
Strak_X_train['Method_2'] = train_xgb_pred
Strak_X_train['Method_3'] = train_gbdt_pred
Strak_X_test = pd.DataFrame()
Strak_X_test['Method_1'] = subA_lgb
Strak_X_test['Method_2'] = subA_xgb
Strak_X_test['Method_3'] = subA_gbdt
## 次级的模型,用LR模型
def build_model_lr(x_train,y_train):
reg_model = linear_model.LinearRegression()
reg_model.fit(x_train,y_train)
return reg_model
model_lr_Stacking = build_model_lr(Strak_X_train,train_y)
## 训练集
train_pre_Stacking = model_lr_Stacking.predict(Strak_X_train)
print('MAE of Stacking-LR:',mean_absolute_error(train_y,train_pre_Stacking))
## 测试集
print('Predict Stacking-LR...')
subA_Stacking = model_lr_Stacking.predict(Strak_X_test)
7081



被折叠的 条评论
为什么被折叠?



