【Kafka】ZK、Kafka以及EFAK的安装、配置

cnskylee

已于 2022-03-30 10:29:35 修改

阅读量4.2k

点赞数

CC 4.0 BY-SA版权

分类专栏： Kafka 文章标签： kafka zookeeper EFAK Kafka eagle

于 2022-03-17 11:05:19 首次发布

本文链接：https://blog.youkuaiyun.com/cnskylee/article/details/123544841

Kafka 专栏收录该内容

4 篇文章

订阅专栏

本文档详细介绍了Zookeeper 3.8.0的配置及启动过程，包括设置clientPort、metricsProvider等，并通过'四字真言'测试验证其运行状态。接着，展示了Kafka 3.1.0的配置，包括broker.id、listeners等，并使用kafka-server-start.sh启动Kafka服务。最后，讲解了EFAK 2.1.0的配置和启动，用于监控Kafka集群，但提到访问EFAK页面可能需要较长时间。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、Zookeeper

版本：apache-zookeeper-3.8.0

配置：

zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/root/zookeeper/data
clientPort=2181
maxClientCnxns=60
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
4lw.commands.whitelist=*

metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpHost=192.168.223.200
metricsProvider.httpPort=17000
metricsProvider.exportJvmInfo=true

zkServer.sh

第 1 处修改：将 243 行clientPortAddress="localhost"修改如下
clientPortAddress="192.168.223.200"

第 2 处修改：将 283 行clientPortAddress="localhost"修改如下
clientPortAddress="192.168.223.200"

启动：

[root@centos-7 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /root/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[root@centos-7 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /root/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

测试“四字真言”

# echo stat| nc 192.168.223.200 2181
Zookeeper version: 3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f, built on 2022-02-25 08:49 UTC
Clients:
 /192.168.223.200:46818[1](queued=0,recved=2,sent=2)
 /192.168.223.200:46830[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46838[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46810[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46828[1](queued=0,recved=4,sent=4)
 /192.168.223.200:46826[1](queued=0,recved=2,sent=2)
 /192.168.223.200:46906[0](queued=0,recved=1,sent=0)
 /192.168.223.200:46824[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46812[1](queued=0,recved=3,sent=3)
 /192.168.223.200:46816[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46834[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46814[1](queued=0,recved=4,sent=4)
 /192.168.223.200:46836[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46820[1](queued=0,recved=4,sent=4)
 /192.168.223.200:46806[1](queued=0,recved=5,sent=5)
 /192.168.223.200:46822[1](queued=0,recved=4,sent=4)
 /192.168.223.200:46808[1](queued=0,recved=3,sent=3)
 /192.168.223.200:46832[1](queued=0,recved=3,sent=3)

Latency min/avg/max: 0/4.8654/66
Received: 76
Sent: 79
Connections: 18
Outstanding: 0
Zxid: 0xd0
Mode: standalone
Node count: 53

其他：

stat 查看状态信息
ruok 查看zookeeper是否启动
dump 列出没有处理的节点，临时节点
conf 查看服务器配置
cons 显示连接到服务端的信息
envi 显示环境变量信息
mntr 查看zk的健康信息
wchs 展示watch的信息
wchc和wchp 显示session的watch信息 path的watch信息

二、Kafka

版本：kafka_2.12-3.1.0

配置：

server.properties

broker.id=0

listeners=PLAINTEXT://192.168.223.200:9092
advertised.listeners=PLAINTEXT://192.168.223.200:9092

num.network.threads=3
num.io.threads=8

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

log.dirs=/root/kafka/logs

num.partitions=1
num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

zookeeper.connect=192.168.223.200:2181
zookeeper.connection.timeout.ms=60000

group.initial.rebalance.delay.ms=0

kafka-server-start.sh

if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
        exit 1
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G -XX:MetaspaceSize=1G -XX:MaxMetaspaceSize=1G"
fi

export JMX_PORT="9010"

EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}

COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

启动脚本：

cd /root/kafka/bin
./kafka-server-start.sh -daemon ../config/server.properties
tail -f ../logs/server.log

三、EFAK

版本：kafka-eagle-bin-2.1.0

配置：

system-config.properties

efak.zk.cluster.alias=cluster1
cluster1.zk.list=192.168.223.200:2181

cluster1.zk.acl.enable=false
cluster1.zk.acl.schema=digest
cluster1.zk.acl.username=test
cluster1.zk.acl.password=test123

cluster1.efak.broker.size=20

kafka.zk.limit.size=16

efak.webui.port=8048

efak.distributed.enable=false
efak.cluster.mode.status=master
efak.worknode.master.host=localhost
efak.worknode.port=8085

cluster1.efak.offset.storage=kafka

cluster1.efak.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi

efak.metrics.charts=true
efak.metrics.retain=15

efak.sql.topic.records.max=5000
efak.sql.topic.preview.records.max=10

efak.topic.token=keadmin

efak.driver=org.sqlite.JDBC
efak.url=jdbc:sqlite:/root/eagle/web/db/ke.db
efak.username=root
efak.password=YOUR_PASSWORD

启动：

# cd /root/eagle/web/bin && ./ke.sh start
   ...
  created: META-INF/maven/
  created: META-INF/maven/org.smartloli.kafka.eagle/
  created: META-INF/maven/org.smartloli.kafka.eagle/efak-web/
 inflated: META-INF/maven/org.smartloli.kafka.eagle/efak-web/pom.xml
 inflated: META-INF/maven/org.smartloli.kafka.eagle/efak-web/pom.properties
[2022-03-16 22:36:04] INFO: Port Progress: [##################################################] | 100%
[2022-03-16 22:36:08] INFO: Config Progress: [##################################################] | 100%
[2022-03-16 22:36:11] INFO: Startup Progress: [##################################################] | 100%
[2022-03-16 22:36:01] INFO: Status Code[0]
[2022-03-16 22:36:01] INFO: [Job done!]
Welcome to
    ______    ______    ___     __ __
   / ____/   / ____/   /   |   / //_/
  / __/     / /_      / /| |  / ,<   
 / /___    / __/     / ___ | / /| |  
/_____/   /_/       /_/  |_|/_/ |_|  
( Eagle For Apache Kafka® )

Version 2.1.0 -- Copyright 2016-2022
*******************************************************************
* EFAK Service has started success.
* Welcome, Now you can visit 'http://192.168.223.200:8048'
* Account:admin ,Password:123456
*******************************************************************
* <Usage> ke.sh [start|status|stop|restart|stats] </Usage>
* <Usage> https://www.kafka-eagle.org/ </Usage>
*******************************************************************

备注：进程启动成功后，访问EFAK的管理页面，需要等很久，不知道什么原因。

EFAK（Eagel for Apache Kafka）的WEB界面：