获取kafka的lag, offset, logsize的shell和python脚本

最新推荐文章于 2025-06-23 21:28:59 发布

weixin_30487701

最新推荐文章于 2025-06-23 21:28:59 发布

阅读量829

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 python shell

原文链接：http://www.cnblogs.com/regit/p/11081071.html

本文介绍了一种使用Python和Shell脚本来查询Kafka消费者组的偏移量、日志大小及延迟的方法。通过解析Kafka消费者组的描述信息，该脚本能够获取并打印指定的统计信息。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

python脚本

#!/usr/bin/env python

import os
import re
import sys

group_id=sys.argv[1]
pn=sys.argv[2]

kafka_ip=os.popen('sudo docker inspect elements_kafka_1 | grep KAFKA_ADVERTISED_HOST_NAME').read()
kafka_ip=re.match('^.*=(.*)",', kafka_ip).group(1)
kafka_port=9092

def kafka_value():

    content=os.popen('sudo docker exec -it elements_kafka_1 /opt/kafka_2.12-2.2.0/bin/kafka-consumer-groups.sh --bootstrap-server kafka_ip:kafka_port --group group_id  --describe | grep -v LAG').read()
    content=content.split()

    length=len(content)
    item_num=length/8

    Offset="0"
    Logsize="0"
    Lag="0"

    for i in range(0, item_num):
        Logsize = content[8*i+3]
        if Logsize == "0":
            continue
        else:
            Offset = content[8*i+2]
            Lag = content[8*i+4]

    if pn in ("lag", "Lag"):
        print(Lag)
    if pn in ("offset", "Offset"):
        print(Offset)
    if pn in ("logsize", "Logsize"):
        print(Logsize)

kafka_value()

shell脚本

#!/bin/bash

#NIC=ens3
#kafka_ip=`ifconfig $NIC | awk '/inet addr:/ {print $2}' | awk -F: '{print $2}'`
kafka_ip=`sudo docker inspect elements_kafka_1 | grep KAFKA_ADVERTISED_HOST_NAME | awk -F "[=,\"]" '{print $3}'`
kafka_port=9092
group_id=$1
pn=$2

sudo docker exec -it elements_kafka_1 \
/opt/kafka_2.12-2.2.0/bin/kafka-consumer-groups.sh --bootstrap-server $kafka_ip:$kafka_port --group $group_id --describe | grep -v LAG \
> /tmp/kafka-tp-${group_id}.info

# modify format problem
sed -i 's/\r//g' /tmp/kafka-tp-${group_id}.info

Offset=0
logsize=0
Lag=0

while read line
do   
    current_logsize=`echo $line | awk '{print $4}'`
    if [ $current_logsize = 0 ];then
      continue     
    else
      current_lag=`echo $line | awk '{print $5}'`
      current_offset=`echo $line | awk '{print $3}'`
      Offset=$((Offset+current_offset))
      logsize=$((logsize+current_logsize))
      Lag=$((Lag+current_lag))
    fi
done < /tmp/kafka-tp-${group_id}.info

case $pn in
    offset|Offset)
    echo $Offset
    ;;
    logsize|logSize)
    echo $logsize
    ;;
    lag|Lag)
    echo $Lag
    ;;
    *)
    echo Error
    ;;
esac