kafka执行工具系列之（一）原生的kafka的脚本

最新推荐文章于 2025-07-18 15:40:32 发布

原创最新推荐文章于 2025-07-18 15:40:32 发布 · 427 阅读

0 ·

CC 4.0 BY-SA版权

kafka 专栏收录该内容

9 篇文章

订阅专栏

原生 kafka 提供的工具脚本：

一、所有工具脚本的位置：

[root@master my_bin]# cd $KAFKA_HOME 
[root@master kafka]# cd bin/
[root@master bin]# ll
总用量 116
-rwxr-xr-x. 1 root root 1052 8月   4 2016 connect-distributed.sh
-rwxr-xr-x. 1 root root 1051 8月   4 2016 connect-standalone.sh
-rwxr-xr-x. 1 root root  861 8月   4 2016 kafka-acls.sh
-rwxr-xr-x. 1 root root  864 8月   4 2016 kafka-configs.sh
-rwxr-xr-x. 1 root root  945 8月   4 2016 kafka-console-consumer.sh
-rwxr-xr-x. 1 root root  944 8月   4 2016 kafka-console-producer.sh
-rwxr-xr-x. 1 root root  871 8月   4 2016 kafka-consumer-groups.sh
-rwxr-xr-x. 1 root root  872 8月   4 2016 kafka-consumer-offset-checker.sh
-rwxr-xr-x. 1 root root  948 8月   4 2016 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 root root  862 8月   4 2016 kafka-mirror-maker.sh
-rwxr-xr-x. 1 root root  886 8月   4 2016 kafka-preferred-replica-election.sh
-rwxr-xr-x. 1 root root  959 8月   4 2016 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 root root  874 8月   4 2016 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 root root  868 8月   4 2016 kafka-replay-log-producer.sh
-rwxr-xr-x. 1 root root  874 8月   4 2016 kafka-replica-verification.sh
-rwxr-xr-x. 1 root root 6358 8月   4 2016 kafka-run-class.sh
-rwxr-xr-x. 1 root root 1364 8月   4 2016 kafka-server-start.sh
-rwxr-xr-x. 1 root root  975 8月   4 2016 kafka-server-stop.sh
-rwxr-xr-x. 1 root root  870 8月   4 2016 kafka-simple-consumer-shell.sh
-rwxr-xr-x. 1 root root  945 8月   4 2016 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 root root  863 8月   4 2016 kafka-topics.sh
-rwxr-xr-x. 1 root root  958 8月   4 2016 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 root root  958 8月   4 2016 kafka-verifiable-producer.sh
drwxr-xr-x. 2 root root 4096 8月   4 2016 windows
-rwxr-xr-x. 1 root root  867 8月   4 2016 zookeeper-security-migration.sh
-rwxr-xr-x. 1 root root 1381 8月   4 2016 zookeeper-server-start.sh
-rwxr-xr-x. 1 root root  978 8月   4 2016 zookeeper-server-stop.sh
-rwxr-xr-x. 1 root root  968 8月   4 2016 zookeeper-shell.sh

二、实际生产使用的命令

1、显示出Consumer的Group、Topic、分区ID、分区对应已经消费的Offset、logSize大小，Lag以及Owner等信息。

KAFKA_HOME/bin/kafka-consumer-offset-checker.sh --zookeeper zk的master的ip:2181  --topic stable-test --group  test-consumer-group

举例:
[root@localhost data]# kafka-consumer-offset-checker --zookeeper localhost :2181  --group test-consumer-group  --topic stable-test
[2017-08-22 19:24:24,222] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Group           Topic                          Pid Offset          logSize         Lag             Owner
test-consumer-group stable-test                    0   601808          601808          0               none
test-consumer-group stable-test                    1   602826          602828          2               none
test-consumer-group stable-test                    2   602136          602136          0               none

在这里插入代码片
/home/migu/kafka/kafka_2.11-0.8.2.2/bin/kafka-consumer-offset-checker.sh --zookeeper zk_master_ip:2181  --topic filter_session_start --group  logProcessor-product|awk '{s +=$4;e +=$5;t +=$6};END {print s"\t"e"\t"t}'    
#后面awk的加上执行条件可看到kafka消费者消费的总的offset数

awk的命令详解