评分卡模型（二数据清洗)

最新推荐文章于 2024-08-27 18:32:22 发布

原创

最新推荐文章于 2024-08-27 18:32:22 发布 · 843 阅读

0 ·

CC 4.0 BY-SA版权

该博客主要介绍了评分卡模型数据清洗的过程，包括删除常数列、检查单值占比、计算坏样本率、处理缺失值等步骤。通过分析特征的缺失情况，对于类别型变量，如果缺失超过80%则删除，数值型变量则根据缺失率填充平均值或进行随机抽样补缺。

# -*- coding: utf-8 -*-
"""
Created on Sun Sep 16 19:04:53 2018

@author: wangxihe
"""

import os
import pandas as pd
import numbers
import numpy as np
import matplotlib.pyplot as plt
#%%
os.chdir(r'E:\spyderwork\评分卡模型\一特征构建')
allData = pd.read_csv('Idx0.csv',header = 0,encoding = 'gbk')
allData.shape
#%%
os.chdir(r'E:\spyderwork\评分卡模型\二特征清洗')
describeDf=allData.describe().T

def MissingCategorial(df,col):
missing_vals = df[col].map(lambda x: int(x!=x))
return sum(missing_vals)*1.0/df.shape[0]

def MissingContinuous(df,col):
missing_vals = df[col].map(lambda x: int(np.isnan(x)))
return sum(missing_vals) * 1.0 / df.shape[0]

#%%
allFeatures = list(allData.columns)
allFeatures.remove('target')
if 'Idx' in allFeatures:
allFeatures.remove('Idx')
allFeatures.remove('ListingInfo')

len(allFeatures)
#%%
#检查是否有常数型变量，并且检查是类别型还是数值型变量

numerical_var = []
for col in allFeatures:
    if len(set(allData[col])) == 1:
        print(' {} :此列为常数所以删除'.format(col))
        del allData[col]
        allFeatures.remove(col)
    else:
        uniq_valid_vals = [i for i in allData[col] if i == i]

        uniq_valid_vals = list(set(uniq_valid_vals))
        if len(uniq_valid_vals) >= 10 and isinstance(uniq_valid_vals[0], numbers.Real):
            numerical_var.append(col)

categorical_var = [i for i in allFeatures if i not in numerical_var]

len(numerical_var)
len(categorical_var)
#%%
#检查变量的最多值的占比情况,以及每个变量中占比最大的值
records_count = allData.shape[0]
col_most