十三：kafka分布式部署

最新推荐文章于 2025-02-17 06:00:00 发布

原创

最新推荐文章于 2025-02-17 06:00:00 发布 · 735 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka

本文介绍了Kafka作为分布式流平台的主要功能，对比了Kafka与Flume的区别，并详细阐述了Kafka的分布式部署步骤，包括Zookeeper的安装、Scala环境配置、Kafka的选型与安装，以及在CDH上的使用和创建Topic的过程。

一：kafka概述：

就一个消息中间件，当前官网叫做：分布式流平台a distributed streaming platform：
A streaming platform has three key capabilities:

1.Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
2.Store streams of records in a fault-tolerant durable way.
3.Process streams of records as they occur.

Kafka is generally used for two broad classes of applications:

1.Building real-time streaming data pipelines that reliably get data between systems or applications
2.Building real-time streaming applications that transform or react to the streams of data

二：kafak对比Flume：

Flume is a distributed, reliable, and available system for efficiently collecting, aggregating, and moving large amounts of log data from many different sources to a centralized data store.

Flume：只有1个进程：含Source Channel Sink
kafka : 有3个进程： producer broker cousumer
生成者服