kafka工具kafka-topic.sh

Kafka Topic管理
本文介绍如何使用kafka-topic.sh工具进行Kafka Topic的基本管理,包括创建、列出、描述及删除等操作,并解释了相关参数的意义。

kafka 工具 kafka-topic.sh

操作

kafka-topic.sh有6种操作:

创建topic – create

[lch@linux129 bin]$ ./kafka-topics.sh --create --zookeeper ip:2181,ip:2181 --replication-factor 2 --partitions 20 --topic TEST
参数说明:
--replication-factor    : 表明 副本的个数, 这里写成2,表明会有两个副本
--partitions              :  表明会有20个partition, 表现在log日志中,就是有20个文件夹,名称以“topic-编号” ,编号从0~19

执行完成后去kafka数据目录下查看,会看到创建了两个文件夹
    test-0
    test-1
    test-2

注: 在leader节点上面才是20个文件夹,如果是从节点上面,可能只有一个也可能没有

列出全部的topic –list命令

[root@tingyun bin]# ./kafka-topics.sh --zookeeper=localhost:2181  --list
__consumer_offsets
crashMessage
mobileActions
mobileActionsWrappedMessages
mobileErrors
mobileErrorsWrappedMessages
mobileInteractionTraces
mobileInteractionTracesWrappedMessages
mobileReportUsers
mobileReportUsersWrappedMessages

列出全部topic的信息 –describe

包括每个分区的leader,副本,isr信息

[root@tingyun bin]# ./kafka-topics.sh --zookeeper=localhost:2181  --describe
Topic:__consumer_offsets    PartitionCount:50   ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed
    Topic: __consumer_offsets   Partition: 0    Leader: 4   Replicas: 4,0,1 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 1    Leader: 0   Replicas: 0,1,2 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 2    Leader: 1   Replicas: 1,2,3 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 3    Leader: 2   Replicas: 2,3,4 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 4    Leader: 3   Replicas: 3,4,0 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 5    Leader: 4   Replicas: 4,1,2 Isr: 1,4,2
    Topic: __consumer_offsets   Partition: 6    Leader: 0   Replicas: 0,2,3 Isr: 0,3,2
    Topic: __consumer_offsets   Partition: 7    Leader: 1   Replicas: 1,3,4 Isr: 1,3,4
    Topic: __consumer_offsets   Partition: 8    Leader: 2   Replicas: 2,4,0 Isr: 0,4,2
    Topic: __consumer_offsets   Partition: 9    Leader: 3   Replicas: 3,0,1 Isr: 0,1,3
    Topic: __consumer_offsets   Partition: 10   Leader: 4   Replicas: 4,2,3 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 11   Leader: 0   Replicas: 0,3,4 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 12   Leader: 0   Replicas: 1,4,0 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 13   Leader: 2   Replicas: 2,0,1 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 14   Leader: 3   Replicas: 3,1,2 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 15   Leader: 4   Replicas: 4,3,0 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 16   Leader: 0   Replicas: 0,4,1 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 17   Leader: 0   Replicas: 1,0,2 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 18   Leader: 2   Replicas: 2,1,3 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 19   Leader: 3   Replicas: 3,2,4 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 20   Leader: 4   Replicas: 4,0,1 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 21   Leader: 0   Replicas: 0,1,2 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 22   Leader: 1   Replicas: 1,2,3 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 23   Leader: 2   Replicas: 2,3,4 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 24   Leader: 3   Replicas: 3,4,0 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 25   Leader: 4   Replicas: 4,1,2 Isr: 1,4,2
    Topic: __consumer_offsets   Partition: 26   Leader: 0   Replicas: 0,2,3 Isr: 0,3,2
    Topic: __consumer_offsets   Partition: 27   Leader: 1   Replicas: 1,3,4 Isr: 1,3,4
    Topic: __consumer_offsets   Partition: 28   Leader: 2   Replicas: 2,4,0 Isr: 0,4,2
    Topic: __consumer_offsets   Partition: 29   Leader: 3   Replicas: 3,0,1 Isr: 0,1,3
    Topic: __consumer_offsets   Partition: 30   Leader: 4   Replicas: 4,2,3 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 31   Leader: 0   Replicas: 0,3,4 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 32   Leader: 0   Replicas: 1,4,0 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 33   Leader: 2   Replicas: 2,0,1 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 34   Leader: 3   Replicas: 3,1,2 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 35   Leader: 4   Replicas: 4,3,0 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 36   Leader: 0   Replicas: 0,4,1 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 37   Leader: 0   Replicas: 1,0,2 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 38   Leader: 2   Replicas: 2,1,3 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 39   Leader: 3   Replicas: 3,2,4 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 40   Leader: 4   Replicas: 4,0,1 Isr: 0,1,4
    Topic: __consumer_offsets   Partition: 41   Leader: 0   Replicas: 0,1,2 Isr: 0,1,2
    Topic: __consumer_offsets   Partition: 42   Leader: 1   Replicas: 1,2,3 Isr: 1,3,2
    Topic: __consumer_offsets   Partition: 43   Leader: 2   Replicas: 2,3,4 Isr: 3,4,2
    Topic: __consumer_offsets   Partition: 44   Leader: 3   Replicas: 3,4,0 Isr: 0,3,4
    Topic: __consumer_offsets   Partition: 45   Leader: 4   Replicas: 4,1,2 Isr: 1,4,2
    Topic: __consumer_offsets   Partition: 46   Leader: 0   Replicas: 0,2,3 Isr: 0,3,2
    Topic: __consumer_offsets   Partition: 47   Leader: 1   Replicas: 1,3,4 Isr: 1,3,4
    Topic: __consumer_offsets   Partition: 48   Leader: 2   Replicas: 2,4,0 Isr: 0,4,2
    Topic: __consumer_offsets   Partition: 49   Leader: 3   Replicas: 3,0,1 Isr: 0,1,3
Topic:TEST  PartitionCount:20   ReplicationFactor:1 Configs:
    Topic: TEST Partition: 0    Leader: 4   Replicas: 4 Isr: 4
    Topic: TEST Partition: 1    Leader: 0   Replicas: 0 Isr: 0
    Topic: TEST Partition: 2    Leader: 1   Replicas: 1 Isr: 1
    Topic: TEST Partition: 3    Leader: 2   Replicas: 2 Isr: 2
    Topic: TEST Partition: 4    Leader: 3   Replicas: 3 Isr: 3
    Topic: TEST Partition: 5    Leader: 4   Replicas: 4 Isr: 4
    Topic: TEST Partition: 6    Leader: 0   Replicas: 0 Isr: 0
    Topic: TEST Partition: 7    Leader: 1   Replicas: 1 Isr: 1
    Topic: TEST Partition: 8    Leader: 2   Replicas: 2 Isr: 2
    Topic: TEST Partition: 9    Leader: 3   Replicas: 3 Isr: 3
    Topic: TEST Partition: 10   Leader: 4   Replicas: 4 Isr: 4
    Topic: TEST Partition: 11   Leader: 0   Replicas: 0 Isr: 0
    Topic: TEST Partition: 12   Leader: 1   Replicas: 1 Isr: 1
    Topic: TEST Partition: 13   Leader: 2   Replicas: 2 Isr: 2
    Topic: TEST Partition: 14   Leader: 3   Replicas: 3 Isr: 3
    Topic: TEST Partition: 15   Leader: 4   Replicas: 4 Isr: 4
    Topic: TEST Partition: 16   Leader: 0   Replicas: 0 Isr: 0
    Topic: TEST Partition: 17   Leader: 1   Replicas: 1 Isr: 1
    Topic: TEST Partition: 18   Leader: 2   Replicas: 2 Isr: 2
    Topic: TEST Partition: 19   Leader: 3   Replicas: 3 Isr: 3

删除topic –delete

 [root@hadoop221 bin]# ./kafka-topics.sh  --zookeeper ip:port  --topic TEST  --delete

topic无法删掉
Topic TEST is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

需要增加配置参数:delete.topic.enable=true

修改topic时配置参数 –alter –config

覆盖已经有topic参数

 [root@hadoop221 bin]# ./kafka-topics.sh --zookeeper zk_host:port  --topic TEST --alter --config flush.messages=1

删除topic级别配置参数

 [root@hadoop221 bin]# ./kafka-topics.sh --zookeeper zk_host:port --alter --topic TEST --delete-config flush.messages=1

修改topic分区数量

 [root@hadoop221 bin]# ./kafka-topics.sh --zookeeper zk_host:port --alter --topic TEST --partitions 6
<think>根据用户的问题,他想要使用`kafka-run-class.sh`来查看Kafka中某个topic的消息数量(即总数据量)。在之前的回答中,我已经介绍了两种方法,其中方法一就是使用`kafka-run-class.sh`(实际上是调用`kafka.tools.GetOffsetShell`)来获取每个分区的偏移量,然后求和得到总消息量。 用户现在明确要求使用`kafka-run-class.sh`,所以我们将重点放在方法一上。 ### 步骤详解 1. **命令格式**: ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server <broker地址> \ --topic <topic名称> \ --time -1 ``` 2. **参数说明**: - `--bootstrap-server`: Kafka broker的地址,例如`localhost:9092`。 - `--topic`: 要查询的topic名称。 - `--time -1`: 表示获取每个分区的最新偏移量(即当前消息总量)。`-2`则表示获取最早的偏移量(即保留的最旧消息位置)。 3. **输出示例**: 假设topic名为`test-topic`,有3个分区,输出可能如下: ``` test-topic:0:100 test-topic:1:150 test-topic:2:200 ``` 每一行格式为:`topic:分区编号:当前偏移量`。 4. **计算总消息量**: 使用`awk`命令对输出进行求和: ```bash ... | awk -F ":" '{sum += $3} END {print "总消息量: " sum}' ``` - `-F ":"`:指定分隔符为冒号。 - `{sum += $3}`:将每行的第三列(即偏移量)累加到变量`sum`中。 - `END {print ...}`:处理完所有行后打印总和。 ### 完整命令 将上述步骤组合起来,得到完整命令: ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server localhost:9092 \ --topic test-topic \ --time -1 | \ awk -F ":" '{sum += $3} END {print "总消息量: " sum}' ``` ### 注意事项 1. **权限问题**:如果Kafka集群启用了安全认证(如SASL/SSL),需要在命令中添加相应的认证参数。例如,使用`--consumer.config`指定配置文件: ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server localhost:9092 \ --topic test-topic \ --time -1 \ --consumer.config /path/to/client.properties ``` 配置文件`client.properties`中通常包含安全认证信息,例如: ``` security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret"; ``` 2. **路径问题**:确保在Kafka的安装目录下执行命令,或者使用绝对路径调用`kafka-run-class.sh`。 3. **环境问题**:如果遇到`java: 未找到`的错误(如引用[4]所述),需要检查并设置`kafka-run-class.sh`中的`JAVA_HOME`。编辑该文件,找到设置`JAVA_HOME`的地方,确保指向正确的JDK安装路径。 4. **偏移量的含义**:偏移量是从0开始计数的,所以分区0的偏移量为100表示有100条消息(偏移量0到99,共100条)。 ### 其他方法 用户引用的内容中提到了使用JMX工具(引用[1])来查看消息入栈速率(`MessageInPerSec`),这是监控消息写入速率的方法,而不是总消息量。所以不适用于当前需求。 ### 相关问题 1. 如何查看Kafka Topic每个分区的消息分布情况? 2. 如何计算Kafka Topic占用的磁盘空间大小? 3. 如何监控Kafka Topic的消息写入速率(TPS)? 4. Kafka中如何查看消费者组的消费进度?</think>### 使用 `kafka-run-class.sh` 查看 Kafka Topic 消息总量 #### 完整操作步骤 1. **执行偏移量查询命令**: ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server <broker地址> \ --topic <topic名称> \ --time -1 ``` - `--bootstrap-server`:Kafka broker地址(如 `localhost:9092`) - `--topic`:目标 topic 名称 - `--time -1`:获取最新偏移量(即当前消息总量)[^3] 2. **计算消息总量**: ```bash # 添加 awk 命令自动求和 ... | awk -F ":" '{sum += $3} END {print "总消息量: " sum}' ``` **完整命令示例**: ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server localhost:9092 \ --topic user_events \ --time -1 | \ awk -F ":" '{sum += $3} END {print "总消息量: " sum}' ``` #### 输出解析 假设输出: ``` user_events:0:1500 user_events:1:2300 user_events:2:1800 总消息量: 5600 ``` - 每行格式:`topic:分区编号:最新偏移量` - **总消息量** = 所有分区偏移量之和: $$ \text{总消息量} = \sum_{i=0}^{n-1} \text{Partition}_i\text{的偏移量} $$ 本例:$1500 + 2300 + 1800 = 5600$ 条消息 #### 常见问题解决 1. **Java 未找到错误**(引用[4]): - 编辑 `kafka-run-class.sh` 文件: ```bash vim bin/kafka-run-class.sh ``` - 修改 `JAVA_HOME` 路径为实际 JDK 安装路径: ```bash # 查找并修改类似行 export JAVA_HOME=/usr/lib/jvm/java-11-openjdk # 改为你的JDK路径 ``` 2. **安全集群认证**(SASL/SSL): ```bash bin/kafka-run-class.sh kafka.tools.GetOffsetShell \ --bootstrap-server secure-broker:9093 \ --topic secured_topic \ --time -1 \ --command-config /path/to/client.properties # 包含认证信息的配置文件 ``` #### 注意事项 1. **偏移量本质**:偏移量从 0 开始计数,分区消息量 = 最新偏移量值 2. **实时性**:结果反映命令执行时刻的快照,非持续监控[^1] 3. **分区影响**:增加分区会导致总量计算波动(需重新分配数据) > 📌 提示:生产环境建议结合监控工具(如 Kafka Manager)实现持续跟踪[^1]。 --- ### 相关问题 1. 如何查看 Kafka Topic 每个分区的消息分布详情? 2. Kafka 中如何计算消息的实际存储大小(GB/MB)? 3. 如何监控 Kafka Topic 的实时消息写入速率? 4. Kafka 消息保留策略如何影响数据量统计结果?
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值