flink sink 2 HDFS工具类

DogTips

于 2021-10-12 10:49:56 发布

阅读量262

点赞数

CC 4.0 BY-SA版权

文章标签： flink

本文链接：https://blog.youkuaiyun.com/DogTips/article/details/120719115

本文介绍了如何在Flink中创建一个自定义的Sink，用于将数据高效地写入HDFS文件系统。通过实现分桶策略，确保数据按特定规则进行分区和存储。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-parquet_2.11</artifactId>
        <version>1.10.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-avro</artifactId>
        <version>1.9.1</version>
    </dependency>

    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-avro</artifactId>
        <version>1.10.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.parquet</groupId>
        <artifactId>parquet-hadoop</artifactId>
        <version>1.10.0</version>
    </dependency>

public class Sink2HdfsUtil {
   

    public static StreamingFileSink<String> sinkRow(Path path) {
   
        StreamingFileSink<String> sinkBuild = StreamingFileSink
                .forRowFormat(path, new SimpleStringEncoder<String>("utf-8"))
                .withBucketAssigner(new MyBucketAssigner())
                .withRollingPolicy(