kafka to hdfs

最新推荐文章于 2024-04-04 22:13:40 发布

bigdataf

最新推荐文章于 2024-04-04 22:13:40 发布

阅读量2.1k

点赞数

文章标签： flume

本文链接：https://blog.youkuaiyun.com/bigdataf/article/details/78596810

版权

本文介绍了一系列与Kafka集成的数据处理工具，包括高性能客户端KaBoom，用于日志收集的Flume，通用数据摄取框架Gobblin，以及将Kafka数据导出到HDFS的方法。这些工具旨在提高数据处理效率，支持从多种数据源提取、转换并加载数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

KaBoom - A High Performance Consumer Client for Kafka
https://github.com/blackberry/KaBoom

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
http://flume.apache.org/

Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop.
http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-HDFS-Ingestion/

The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.
https://docs.confluent.io/current/connect/connect-hdfs/docs/hdfs_connector.html

This is a hadoop job for incremental loading of kafka topics into hdfs
https://github.com/amient/kafka-hadoop-loader

https://www.cnblogs.com/zdfjf/p/5646525.html