如何做特征选择？

1 参考链接

Topic 6. Feature Engineering and Feature Selection
https://www.kaggle.com/kashnitsky/topic-6-feature-engineering-and-feature-selection

结合Scikit-learn介绍几种常用的特征选择方法
https://blog.youkuaiyun.com/LY_ysys629/article/details/53641569

2 原因

• 越多的数据，越高的复杂度
• 噪声特征造成过拟合

3 鉴别

• 统计：特征不改变，相当于没有提供任何信息。所以可以说，低方差的特征比高方差的特征差，所以可以考虑去掉方差阈值少于一定阈值的特征。

from sklearn.feature_selection import VarianceThreshold
VarianceThreshold(.7).fit_transform(x_data_generated).shape

• 分类统计

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

x_data_kbest = SelectKBest(f_classif, k=5).fit_transform(x_data_generated, y_data_generated)
x_data_varth = VarianceThreshold(.9).fit_transform(x_data_generated)

cross_val_score(LogisticRegression(), x_data_generated, y_data_generated, scoring='neg_log_loss').mean()
-0.4085819819399657

cross_val_score(LogisticRegression(), x_data_kbest, y_data_generated, scoring='neg_log_loss').mean()
-0.3391459274373541

cross_val_score(LogisticRegression(), x_data_varth, y_data_generated, scoring='neg_log_loss').mean()
-0.37782243684017525

可以看出处理后的特征分数比较高。

• 模型来选择特征，利用Random Forest或者linear model这些简单的模型容易使弱特征无效，所以没比较将这些特征放到复杂的模型中去。

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

x_data_generated, y_data_generated = make_classification()

pipe = make_pipeline(SelectFromModel(estimator=RandomForestClassifier()), LogisticRegression())

lr = LogisticRegression()
rf = RandomForestClassifier()

print(cross_val_score(lr, x_data_generated, y_data_generated, scoring='neg_log_loss').mean())
print(cross_val_score(rf, x_data_generated, y_data_generated, scoring='neg_log_loss').mean())
print(cross_val_score(pipe, x_data_generated, y_data_generated, scoring='neg_log_loss').mean()
-0.3568462635817046
-0.9547234045086143
-0.28555118731986034

当然上述方法也可能使结果变差

• 暴力搜索：以所有特征的子集训练，得到结果，然后重复不同的子集，通过比较模型的结果来确定最优的feature，当然搜索所有空间太耗时，可以一个相对较小的N，迭代选择N个features的组合，来选择特定的最优组合，然后再考虑添加新的特征，来优化，直到特征没有显著提升效果。

from mlxtend.feature_selection import SequentialFeatureSelector

selector = SequentialFeatureSelector(LogisticRegression(), scoring='neg_log_loss', 
                                     verbose=2, k_features=3, forward=False, n_jobs=-1)

selector.fit(x_data, y_data)