读取文件的代码

一个关注技术成长的小博主

于 2021-03-29 08:03:41 发布

阅读量522

点赞数

文章标签： python

本文链接：https://blog.youkuaiyun.com/qq_43380844/article/details/115292009

版权

该代码实现从数据集中获取训练数据，并使用RandomUnderSampler和RandomOverSampler处理类别不平衡问题，通过调整不同类别的样本数量达到平衡。最终将处理后的数据集存储为新的列表。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

def get_data_train(filepath):
    df = pd.read_csv(filepath,sep='\t')
    questions = df['Question']
    sentences = df['Sentence']
    labels = df['Label']
    ids = df['QuestionID']
    QuestionID = []
    data_list = []
    data_x = []
    data_y = []
    for i ,j in zip(df['QuestionID'], labels):
        if j==1 and i not in QuestionID:
            QuestionID.append(i)
    #print(QuestionID)
    for id in QuestionID:
        for index in range(len(ids)):
            if id==ids[index]:
                data_x.append((questions[index],sentences[index]))
                data_y.append(labels[index])


                #data_list.append((questions[index],sentences[index],to_categorical(labels[index], 2)))
    ros_under = RandomUnderSampler(sampling_strategy={0:3000}, random_state=0)
    x_ros_under, y_ros_under = ros_under.fit_resample(np.array(data_x), np.array(data_y).reshape(-1, 1))
    ros_over = RandomOverSampler(sampling_strategy={1:3000}, random_state=0)
    x_ros, y_ros = ros_over.fit_resample(x_ros_under,y_ros_under)
    for i,j in zip(x_ros.tolist(), y_ros.tolist()):
        data_list.append((i[0], i[1], to_categorical(j, 2)))

    data_list = np.array(data_list)
    return data_list