kafka to hdfs

最新推荐文章于 2024-04-04 22:13:40 发布

原创最新推荐文章于 2024-04-04 22:13:40 发布 · 2.1k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#flume

本文介绍了一系列与Kafka集成的数据处理工具，包括高性能客户端KaBoom，用于日志收集的Flume，通用数据摄取框架Gobblin，以及将Kafka数据导出到HDFS的方法。这些工具旨在提高数据处理效率，支持从多种数据源提取、转换并加载数据。

KaBoom - A High Performance Consumer Client for Kafka
https://github.com/blackberry/KaBoom

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
http://flume.apache.org/

Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop.
http://gobblin.readthedocs.io/en/latest/case-studies/Kafka-HDFS-Ingestion/

The HDFS connector allows you to export data from Kafka topics to HDFS files in a variety of formats and integrates with Hive to make data immediately available for querying with HiveQL.
https://docs.confluent.io/current/connect/connect-hdfs/docs/hdfs_connector.html

This is a hadoop job for incremental loading of kafka topics into hdfs
https://github.com/amient/kafka-hadoop-loader

https://www.cnblogs.com/zdfjf/p/5646525.html