Kafka基本命令

最新推荐文章于 2023-09-11 14:38:32 发布

zhang_csdn999

最新推荐文章于 2023-09-11 14:38:32 发布

阅读量207

点赞数

文章标签： hadoop kafka big data

本文链接：https://blog.youkuaiyun.com/zhang_csdn999/article/details/121040619

版权

本文介绍了Kafka的基本命令操作，包括启动Kafka和Zookeeper，查看及创建topic，使用生产者和消费者，以及删除topic。在创建topic时，需要注意复制因子不能超过可用broker数量。通过kafka-console-producer.sh和kafka-console-consumer.sh命令进行消息生产和消费。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

启动

zk.sh start
kafka.sh start

查看topic列表

[hadoop@hadoop1 ~]$ kafka-topics.sh --zookeeper hadoop1:2181 --list

__consumer_offsets

创建topic

[hadoop@hadoop1 logs]$ kafka-topics.sh --zookeeper hadoop1:2181 --create --replication-factor 3 --partitions 2 --topic mytopic1

Created topic "mytopic1".

//查看主题

[hadoop@hadoop1 ~]$ cd /opt/module/kafka/logs

[hadoop@hadoop1 logs]$ cd mytopic1-0

[hadoop@hadoop1 logs]$ kafka-topics.sh --zookeeper hadoop1:2181 --create --replication-factor 2 --partitions 3 --topic mytopic2

Created topic "mytopic2".

[hadoop@hadoop1 logs]$ kafka-topics.sh --zookeeper hadoop1:2181 --create --replication-factor 4 --partitions 2 --topic mytopic3

Error while executing topic command : replication factor: 4 larger than available brokers: 3

[2020-10-16 22:27:48,002] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 4 larger than available brokers: 3

(kafka.admin.TopicCommand$)

查看topic

[hadoop@hadoop1 logs]$ kafka-topics.sh --zookeeper hadoop1:2181 --describe --topic mytopic2

Topic:mytopic2 PartitionCount:3 ReplicationFactor:2 Configs:

Topic: mytopic2 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0

Topic: mytopic2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1

Topic: mytopic2 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2

生产者

[hadoop@hadoop1 logs]$ kafka-console-producer.sh --broker-list hadoop1:9092 --topic mytopic2

>hello

>world

消费者

[hadoop@hadoop1 ~]$ kafka-console-consumer.sh --zookeeper hadoop1:2181 --topic mytopic2

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

[hadoop@hadoop1 ~]$ kafka-console-consumer.sh --bootstrap-server hadoop1:9092 --topic mytopic2 --from-beginning

world

hello

设定组名

[hadoop@hadoop1 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytest --consumer-property group.id=g1

删除topic

[hadoop@hadoop1 ~]$ kafka-topics.sh --zookeeper hadoop1:2181 --delete --topic mytopic1

Topic mytopic1 is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

[hadoop@hadoop1 ~]$ kafka-topics.sh --zookeeper hadoop1:2181 --delete --topic mytopic2

Topic mytopic2 is marked for deletion.

Note: This will have no impact if delete.topic.enable is not set to true.

[hadoop@hadoop1 ~]$ kafka-console-consumer.sh --bootstrap-server hadoop1:9092 --topic __consumer_offsets --from-beginning --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"