kafka常用命令

最新推荐文章于 2025-03-13 17:41:35 发布

原创最新推荐文章于 2025-03-13 17:41:35 发布 · 2.5k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #大数据

常用命令专栏收录该内容

20 篇文章

订阅专栏

本文档详细介绍了如何在Kafka集群中创建和管理Topic，包括使用`--bootstrap-server`或`--zookeeper`参数指定服务器，创建、列出、描述和消费Topic，以及设置复制因子和分区数。同时，展示了不同场景下使用`kafka-topics.sh`和`kafka-console-consumer.sh`命令进行操作的例子。

集群搭建https://www.cnblogs.com/linyouyi/p/11369015.html

Only one of --bootstrap-server or --zookeeper must be specified
–bootstrap-server spark01:9092
–zookeeper spark01:2181

创建topic
bin/kafka-topics.sh --create --bootstrap-server spark01:9092 --replication-factor 3 --partitions 2 --topic linyouyi
./kafka-topics.sh --zookeeper spark01:2181 --topic shop_sync.dbo.Park --create --replication-factor 1 --partitions 3
./kafka-topics.sh --zookeeper spark01:2181 --topic mongo_shake_test --create --replication-factor 1 --partitions 3
./kafka-topics.sh --zookeeper cdh01:2181,cdh02:2181,cdh03:2181 --topic sink_track --create --replication-factor 1 --partitions 3

/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin
./kafka-topics --create --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181 --replication-factor 1 --partitions 3 --topic sink_trace_new

列出所有topic
bin/kafka-topics.sh --list --bootstrap-server spark01:9092
bin/kafka-topics.sh --list --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181

kafka-topics --list --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181

向topic写数据
bin/kafka-console-producer.sh --broker-list spark01:9092 --topic linyouyi>This is a message

实时消费数据
cd /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/kafka/bin
./kafka-console-consumer.sh --bootstrap-server cdh02:9092 --topic user_behavior / sink_track
./kafka-console-consumer.sh --bootstrap-server cdh01:2181,cdh02:2181,cdh03:2181 --topic sink_track / sink_track
./kafka-console-consumer.sh --bootstrap-server bm01:9092,bm02:9092,bm03:9092 --topic sink_track

从头消费数据
bin/kafka-console-consumer.sh --bootstrap-server spark01:9092 --topic user_topic --from-beginning | grep BehaviorDate

bin/kafka-console-consumer.sh --bootstrap-server spark01:9092 --topic sink_track --from-beginning | grep BehaviorDate

kafka-console-consumer --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092 --topic sink_trace_new --from-beginning

查看topic情况
bin/kafka-topics.sh --describe --bootstrap-server hadoop02:9092,hadoop02:9092,hadoop03:9092 --topic sink_track

kafka-topics --describe --bootstrap-server hadoop01:9092,hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092 --topic shop-cdc.dbo.SaleBill

kafka-topics --describe --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181,hadoop04:2181,hadoop05:2181 --topic shop-cdc.dbo.SaleBill

./kafka-topics.sh --describe --zookeeper hadoop01:2181,hadoop02:2181,hadoop03:2181 --topic sink_track
./kafka-topics.sh --describe --zookeeper bm01:2181,bm02:2181,bm03:2181 --topic sink_track
./kafka-topics.sh --describe --zookeeper cdh01:2181,cdh02:2181,cdh03:2181 --topic sink_track