flink消费kafka数据直接到hdfs

小小聪

于 2019-03-25 11:03:41 发布

阅读量1.8k

点赞数

分类专栏： flink

本文链接：https://blog.youkuaiyun.com/servletwjx/article/details/88788279

版权

本文详细介绍了如何使用Apache Flink从Kafka topic实时消费数据，并将这些数据无缝写入HDFS文件系统的过程。通过实例演示配置Flink连接器，实现数据流处理与存储的高效集成。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.fs.SequenceFileWriter;
import org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink;
import org.apache.flink.streaming.connectors.fs.bucketing.DateTimeBucketer;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010;
import org.apache.flink.util.Collector;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;

import java.util.Properties;

public class ReadingToKafka {

    public static void main(String[] args) throws Ex

最低0.47元/天解锁文章