利用model_selection中的train_test_split对整个dataset进行切分

本文介绍了如何使用Python的sklearn库中的train_test_split函数来进行数据集的划分,以支持交叉验证过程。通过具体实例展示了如何将数据集分为训练集和测试集,并解释了随机状态和测试集比例等参数的作用。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

交叉验证:评估模型的表现
1.使用train_test_split可以对训练和测试集进行快速的切分
在切分之前该函数参数中的shuffle的default = True,默认
的会对数据进行洗牌之后再切分

import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5,2)),range(5)

可以看到X是一个5*2的结构也就是5个samples,2个features
y是对应的label对于samples的数量同样为5个

print('X',X)
print('y', y)
X [[0 1]
 [2 3]
 [4 5]
 [6 7]
 [8 9]]
y range(0, 5)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2
                                                   ,random_state = 42)

可以使用model_selection模块中的train_test_split来对整个
数据集进行切分,第一个参数为n_samplesm_features,第二
个参数为labels,第三个参数为测试集所占所有数据的比例,随
机的状态为,如上参数所示test_size = 0.2 那么测试集大小应
当为n_samples
test_size = 1对应的训练集则为4

X_train
array([[8, 9],
       [4, 5],
       [0, 1],
       [6, 7]])
y_train
[4, 2, 0, 3]
X_test
array([[2, 3]])

``` # 数据集字段示例(共87个特征) features = [ 'Flow Duration', 'Total Fwd Packets', 'Total Bwd Packets', 'Flow Bytes/s', 'Packet Length Mean', 'ACK Flag Count', 'Init_Win_bytes_forward', 'Label' # 标签:Benign或DDoS ] import pandas as pd from sklearn.model_selection import train_test_split # 加载数据集(假设已下载并解压为train.csv) data = pd.read_csv('train.csv') # 清洗数据:删除空值、重复项 data = data.dropna().drop_duplicates() # 标签编码:Benign->0, DDoS->1 data['Label'] = data['Label'].apply(lambda x: 0 if x == 'Benign' else 1) # 划分训练集和测试集(8:2) X = data.drop('Label', axis=1) y = data['Label'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # 通过随机森林评估特征重要性 from sklearn.ensemble import RandomForestClassifier rf = RandomForestClassifier(n_estimators=100) rf.fit(X_train, y_train) # 可视化Top10重要特征 import matplotlib.pyplot as plt feature_importances = pd.Series(rf.feature_importances_, index=X.columns) top_features = feature_importances.nlargest(10) top_features.plot(kind='barh', title='Feature Importance') plt.show() # 选取Top10特征重构数据集 selected_features = top_features.index.tolist() X_train = X_train[selected_features] X_test = X_test[selected_features] from xgboost import XGBClassifier from sklearn.metrics import classification_report, roc_auc_score # 初始化模型 model = XGBClassifier( n_estimators=200, max_depth=5, learning_rate=0.1, subsample=0.8 ) # 训练与预测 model.fit(X_train, y_train) y_pred = model.predict(X_test) # 输出评估报告 print(classification_report(y_test, y_pred)) print(f"ROC-AUC Score: {roc_auc_score(y_test, y_pred):.4f}") from flask import Flask, request, jsonify import joblib app = Flask(__name__) model = joblib.load('ddos_model.pkl') # 加载保存的模型 @app.route('/detect', methods=['POST']) def detect(): data = request.json features = [data[f] for f in selected_features] # 使用之前选定的特征 prediction = model.predict([features])[0] return jsonify({'is_ddos': int(prediction)}) if __name__ == '__main__': app.run(port=5000)```Traceback (most recent call last): File "E:\documents\Graduation Design\Code\123.py", line 21, in <module> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\python\Lib\site-packages\sklearn\utils\_param_validation.py", line 216, in wrapper return func(*args, **kwargs) File "E:\python\Lib\site-packages\sklearn\model_selection\_split.py", line 2851, in train_test_split n_train, n_test = _validate_shuffle_split( ~~~~~~~~~~~~~~~~~~~~~~~^ n_samples, test_size, train_size, default_test_size=0.25 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "E:\python\Lib\site-packages\sklearn\model_selection\_split.py", line 2481, in _validate_shuffle_split raise ValueError( ...<3 lines>... ) ValueError: With n_samples=0, test_size=0.2 and train_size=None, the resulting train set will be empty. Adjust any of the aforementioned parameters.
最新发布
04-02
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值