Machine learning series: Handling Missing Values

The method cann’t deal with ‘object’ type data

Explore missing values

missing_val_count_by_column = (data.isnull().sum())
print(missing_val_count_by_column[missing_val_count_by_column > 0])

A simple option: Drop columns with Missing values

## drop data
data_without_missing_values = original_data.dropna(axis=1)
## drop the same columns in both training and test data
cols_with_missing = [col for col in training_data.columns 
                                 if training_data[col].isnull().any()]
redued_training_data = trainingdata.drop(cols_with_missing, axis=1)
reduced_test_data = test_data.drop(cols_with_missing, axis=1)
## The loses access to this information when the column is dropped. Also, if test data has missing values in places where training data did not, this will result in an error.

A better option: Imputation

Imputation fills in the missing value with some number.

from sklearn.impute import SimpleImputer
my_imputer = SimpleImputer()
data_with_imputed_values = pd.DataFrame(my_imputer.fit_transform(original_data))
## my_imputer.fit_transform(original_data) will be a ndarray of ndarrays and will not be structured like Pandas DataFrame, so use pd.DataFrame .But this will lose the column titles, since the order of columns does not change, so we use code below to get the title.
data_with_imputed_values.columns = original_data.columns

function

def handle_missing_value(original_data):
    my_imputer = SimpleImputer()
    new_data = pd.DataFrame(my_imputer.fit_transform(original_data))
    new_data.columns = original_data.columns
    return new_data

An extension to imputation

## make copy to avoid changing original data(when imputing)
new_data = original_data.copy()
## make new columns indicating what will be imputed. These new columns indicate if the value was originally recorded or added in later(by imputation here)
cols_with_missing = (col for col in new_data.columns 
		if new_data[col].isnull().any())
for col in cols_with_missing:
    new_data[col + '_was_missing'] = new_data[col].isnull()

## imputation
my_imputer = SimpleImputer()
new_data = pd.DataFrame(my_imputer.fit_transform(new_data))
new_data.columns = original_data.columns

## In some cases this approach will meaningfully improve results. In other cases, it doesn't help at all.

function

def handle_missing_value(original_data):
    
    new_data = original_data.copy()
    cols_with_missing = (col for col in new_data.columns if new_data[col].isnull().any())
    for col in cols_with_missing:
        new_data[col + '_was_missing'] = new_data[col].isnull()
    new_data_columns = new_data.columns
    ## imputation
    my_imputer = SimpleImputer()
    new_data = pd.DataFrame(my_imputer.fit_transform(new_data))
    new_data.columns = new_data_columns
    
    return new_data