集成学习6——Blending与Stacking

github地址：DataScicence
集成学习5-Xgboost原理与调参
集成学习4-前向分步算法与GBDT-原理与案例
集成学习3-Boosting的原理和案例
集成学习2-bagging的原理与案例分析
集成学习1-投票法的原理和案例分析

Blending

原理

步骤：

• 将数据划分为Train_data、Validate_data、Test_data三部分
• 第一层模型：
  • 使用多个base模型在Train_data上进行训练，得到多个模型$M^k$
  • 分别将Validate_data,Test_data 输入模型，得到预测标签$A^k,B^k$
• 第二层模型：
  • 使用验证集上的预测标签$A^k$作为输入，真实标签作为输出，在meta模型上训练得到模型$N$
  • 将$B^k$作为输入，输入模型N种，得到最终在测试集上预测标签$T$
• 评价模型效果

示例

import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt 
import seaborn as sns

导入数据

from sklearn.datasets import load_iris

data = load_iris()

X = data.data[:,1:3]
y = data.target
print(X.shape,y.shape)

(150, 2) (150,)

数据划分

from sklearn.model_selection import train_test_split

X_train_,X_test,y_train_,y_test = train_test_split(X,y,test_size = 0.2)

X_train,X_val,y_train,y_val = train_test_split(X_train_,y_train_)

print(X_train.shape,X_val.shape,X_test.shape)

(90, 2) (30, 2) (30, 2)

第一层分类器

from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier

# 模型M^k
clfs = [SVC(probability = True),DecisionTreeClassifier(),KNeighborsClassifier()]

val_features = np.zeros((X_val.shape[0],len(clfs)))  # A^k
test_features = np.zeros((X_test.shape[0],len(clfs)))  #B^k

for i,clf in enumerate(clfs):
    clf.fit(X_train,y_train)
    val_feature = clf.predict_proba(X_val)[:, 1]
    test_feature = clf.predict_proba(X_test)[:,1]
    val_features[:,i] = val_feature
    test_features[:,i] = test_feature

第二层分类器

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()

#模型N
lr.fit(val_features,y_val)
# 输出预测的结果
from sklearn.model_selection import cross_val_score
cross_val_score(lr,test_features,y_test,cv=5)

array([0.83333333, 0.66666667, 0.66666667, 0.83333333, 0.66666667])

绘制决策边界

这里的Blending边界并不准确，因为lr模型的输入是第一层模型的预测结果，而不是训练数据

from mlxtend.plotting import plot_decision_regions
import matplotlib.gridspec as gridspec
import itertools
clfs.append(lr)
gs = gridspec.GridSpec(2, 2)
fig = plt.figure(figsize=(10,8))
for clf, lab, grd in zip(clfs, 
                         ['SVC', 
                          'DT', 
                          'KNN',
                          'Blending'],
                          itertools.product([0, 1], repeat=2)):
    clf.fit(X, y)
    ax = plt.subplot(gs[grd[0], grd[1]])
    fig = plot_decision_regions(X=X, y=y, clf=clf)
    plt.title(lab)
plt.show()

stacking

原理

在Blending中，我们训练模型只使用了一部分数据，数据并没有得到充分地利用，因此可以利用交叉验证的方式，将所有的数据都利用上

步骤：

• 将数据切分为训练集Train_data、测试集Test_data
• 第一层模型
  • 在m个模型M上分别使用K折交叉验证在Train_data上训练，并在K个验证集上进行预测，得到预测标签$A^k(将k个结果堆叠，与训练数据的长度相同)$，在测试集上进行预测，得到预测标签$B^k(将K个结果加权平均，与测试集长度相同)$
  • m个模型的结果纵向堆叠得到$A^{m\times k}、B^{m\times k}$
• 第二层模型
  • 使用$A^{m\times k}$作为输入，在Meta模型N上训练，得到模型N
  • 将$B^{m\times k}$输入模型，得到最终预测结果
• 评价模型效果

示例

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB 
from sklearn.ensemble import RandomForestClassifier
from mlxtend.classifier import StackingCVClassifier

模型训练

clf1 = KNeighborsClassifier(n_neighbors=1)
clf2 = RandomForestClassifier()
clf3 = GaussianNB()
lr = LogisticRegression()


clf_stacking = StackingCVClassifier(classifiers=[clf1,clf2,clf3],
                                   meta_classifier=lr,
                                   cv=5)
for clf, label in zip([clf1, clf2, clf3, clf_stacking], ['KNN', 'Random Forest', 'Naive Bayes','StackingClassifier']):
    scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')
    print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

Accuracy: 0.91 (+/- 0.01) [KNN]
Accuracy: 0.95 (+/- 0.01) [Random Forest]
Accuracy: 0.91 (+/- 0.02) [Naive Bayes]
Accuracy: 0.96 (+/- 0.02) [StackingClassifier]

决策边界

from mlxtend.plotting import plot_decision_regions
from matplotlib import gridspec
import itertools

gs = gridspec.GridSpec(2,2)
fig =  plt.figure(figsize=(10,8))

for clf,lab,grd in zip([clf1,clf2,clf3,clf_stacking],
                      ['KNN', 
                          'Random Forest', 
                          'Naive Bayes',
                          'StackingCVClassifier'],
                       itertools.product([0,1],repeat=2)
                      ):
    clf.fit(X,y)
    ax = plt.subplot(gs[grd[0],grd[1]])
    fig = plot_decision_regions(X=X,y=y,clf=clf)
    plt.title(lab)
plt.show()

stacking模型的网格调参

from sklearn.model_selection import GridSearchCV

params = {'kneighborsclassifier__n_neighbors': [1, 5],
          'randomforestclassifier__n_estimators': [10, 50],
          'meta_classifier__C': [0.1, 10.0]}

grid = GridSearchCV(estimator=clf_stacking, 
                    param_grid=params, 
                    cv=5,
                    refit=True)
grid.fit(X, y)
print('Best parameters: %s' % grid.best_params_)
print('Accuracy: %.2f' % grid.best_score_)

Best parameters: {'kneighborsclassifier__n_neighbors': 5, 'meta_classifier__C': 0.1, 'randomforestclassifier__n_estimators': 50}
Accuracy: 0.95

clf_stacking.score(X,y)

0.9866666666666667

基模型使用不同的特征子集

from sklearn.pipeline import make_pipeline
from mlxtend.feature_selection import ColumnSelector
iris = load_iris()
X = iris.data
y = iris.target

pipe1 = make_pipeline(ColumnSelector(cols=(0, 2)),  # 选择第0,2列
                      LogisticRegression())
pipe2 = make_pipeline(ColumnSelector(cols=(1, 2, 3)),  # 选择第1,2,3列
                      LogisticRegression())

sclf = StackingCVClassifier(classifiers=[pipe1, pipe2], 
                            meta_classifier=LogisticRegression(),
                            random_state=42)

sclf.fit(X, y)

StackingCVClassifier(classifiers=[Pipeline(steps=[('columnselector',
                                                   ColumnSelector(cols=(0, 2))),
                                                  ('logisticregression',
                                                   LogisticRegression())]),
                                  Pipeline(steps=[('columnselector',
                                                   ColumnSelector(cols=(1, 2,
                                                                        3))),
                                                  ('logisticregression',
                                                   LogisticRegression())])],
                     meta_classifier=LogisticRegression(), random_state=42)

from sklearn.model_selection import cross_val_score
cross_val_score(sclf,X,y,cv=5)

array([0.96666667, 0.96666667, 0.9       , 0.96666667, 1.        ])