Flink入门(九)batch读csv文件

最新推荐文章于 2024-08-09 11:28:10 发布

小白鸽

最新推荐文章于 2024-08-09 11:28:10 发布

阅读量4.1k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： Flink 文章标签： flink-batch csv

本文链接：https://blog.youkuaiyun.com/baifanwudi/article/details/89848176

Flink 专栏收录该内容

23 篇文章

订阅专栏

博客介绍了使用Flink进行批处理时读取CSV数据的相关内容。采用POJO格式，给出具体代码，并对参数进行说明，如设置分隔符、忽略首行、选取列等，还提到字段名需与POJO类里的字段名对应，最后展示了运行结果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

原数据结构

parent_order_no,member_id,union_id,extra_cost,create_date
TW168D72A322BWWAA6KGP1608,100001608,ohmdTt8m7-j8VXCvHhG2LFVTlNrs,40.0,2019-03-08
TW1694B5C000BPDWOJG5D2464,100002464,ohmdTtyEE9Rh4T-ZPq00IIQiS-uc,40.0,2019-03-08
TW168BFF951C1D7LHWOYH2738,100002738,ohmdTt8BbL0deC-dMGI_pzRPQcOo,100.0,2019-03-08
TW1690D91984B0KK66OBI3068,100003068,ohmdTtyVqnpghR9BFH3I6PSJn8T0,40.0,2019-03-08
TW1693E6E9677T9ELGDNN4670,100004670,ohmdTt_IyzpUxCKtqHINIKJ5MkJc,40.0,2019-03-08
TW1694D338F11PRGPHMOT5809,100005809,ohmdTt2K6SH02gf_rbwE4soT90F8,40.0,2019-03-08
TW1692FEE7B06WAQSUKCC6805,100006805,ohmdTtz2NvntmnsxWAaBWNkmsDmo,40.0,2019-03-08
TW168C2A9C33DE0UKNOLF0805,100010805,ohmdTt-waG_aRzV59di0483nt7aA,40.0,2019-03-08
TW169147C279F1JJ07PPF1805,100011805,ohmdTtzpIPpKkWDVaRsHgO27emyM,120.0,2019-03-08
TW168C5C80F9ECDIWUGIB3319,100013319,ohmdTt84z3BI09Mqj7cgyZQdl0ew,80.0,2019-03-08
TW168EAB7937DXLIRLZMI3357,100013357,ohmdTt8er7xE8vHdw2POeAem_aHI,20.0,2019-03-08
TW1695587BDA0FMP1HTPH3388,100013388,ohmdTt2VFTfIoVx9u_8xuS9Whpts,20.0,2019-03-08
TW168CF81E752XRKR1Q9J5912,100015912,ohmdTtzbP0Hae4scF0DyX6TodZrw,40.0,2019-03-08
TW16936525397V3ITOUMF9886,100019886,ohmdTt6ZhQPPxMk3O-YplvyzcW9M,80.0,2019-03-08
TW169317FF378SXNIYHMU0000,100020000,ohmdTt59sosF9GhoNaHE8PCtZ_T0,40.0,2019-03-08
TW168D036297DSSK9HQYR2838,100022838,ohmdTt59CMK_jpsWu8Wiuh33wZEY,40.0,2019-03-08
TW169132B3B189UA8GGYO9901,100029901,ohmdTt1wYXoPFQetSEwXJQ5l-WMI,30.0,2019-03-08
TW169146C05C1E3O0TZ4D2737,100032737,ohmdTt8yDesjPRG_bf81xPamepJE,20.0,2019-03-08

使用POJO格式

import lombok.Data;

@Data
public class OrderCost {

    private String parentOrderNo;

    private String memberId;

    private String unionId;

//    private String extraCost;

    private String createTime;
}

具体代码如下

        final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

        DataSet<OrderCost> originData=env.readCsvFile("D:\\data\\csv\\stats_order_total_xcost.csv").fieldDelimiter(",")
                .ignoreFirstLine()
                .includeFields(true,true,true,false,true)
                .pojoType(OrderCost.class,"parentOrderNo","memberId","unionId","createTime");
        originData.print();

参数说明：
fieldDelimiter设置分隔符，默认的是","
ignoreFirstLine忽略第一行
includeFields是设置选取哪几列，我这里是第四列不选取的。
pojoType和后面字段名，就是对应列。字段名是不需要对应原始文件header字段名，
但必须与POJO例如OrderCost.class里的字段名一一对应

运行结果如下

OrderCost(parentOrderNo=TW1694C2FA7795LJ6Z3MP5403, memberId=108345403, unionId=ohmdTt9hxYFzeDiap2kkML15RPnc, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1692DCC68EEQ51AEUDY5405, memberId=108345405, unionId=ohmdTt5f2BLeWHtiWs92zxSgxyaY, createTime=2019-03-08)
OrderCost(parentOrderNo=TW168DA84E8B97F867BPX0260, memberId=108350260, unionId=ohmdTt9Al5BJwb2wC-0_ekqstfpU, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1690B10C434UL3YGJON2781, memberId=108352781, unionId=ohmdTt5zjflbGb8O39g9gzuYnBA0, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1694D3A00FFFSMURKZ63109, memberId=108353109, unionId=ohmdTt4JkDJ271MELW0sa8NyvLxM, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1694E18A708FZC2AQ9R4300, memberId=108354300, unionId=ohmdTt6myDMZY1D2OdN9y7fSUyhA, createTime=2019-03-08)
OrderCost(parentOrderNo=TW169038089DDYMAR9WDU8675, memberId=108358675, unionId=ohmdTt0K6dcYTrTTyv8d0PVpDDe4, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1691ABD3503PIFNHB0D9997, memberId=108359997, unionId=ohmdTt0Vh4hFzgcu7XobHGRkmJpg, createTime=2019-03-08)
OrderCost(parentOrderNo=TW168F42E09667F4B7HSV1407, memberId=108361407, unionId=ohmdTt6BGR3eHA7eI2WR94OcaNQE, createTime=2019-03-08)
OrderCost(parentOrderNo=TW169501012ECPAUYMYA41879, memberId=108361879, unionId=ohmdTtyfyJyLRyr13vz1d-lAhuCw, createTime=2019-03-08)
OrderCost(parentOrderNo=TW1694CCF173CLBBUNB2U3369, memberId=108363369, unionId=ohmdTt3cii2phf66SWY7ikSIQvRY, createTime=2019-03-08)