kafka_2.12集群布署

最新推荐文章于 2024-04-19 11:51:27 发布

原创最新推荐文章于 2024-04-19 11:51:27 发布 · 709 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#kafka

Linux日常运维专栏收录该内容

9 篇文章

订阅专栏

下载kafka：

https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz

配置JDK环境：

echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile

echo "export CLASSPATH=\$JAVA_HOME/lib:\$CLASSPATH" >> /etc/profile

echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile

source /etc/profile

tar -xf kafka_2.12-2.5.0.tgz

mv kafka_2.12-2.5.0 /usr/local/kafka

cd /usr/local/kafka/

1、zookeeper集群配置

mkdir -p /data/zookeeper

修改各节点zookeeper.properties配置文件如下：

192.168.199.140

创建集群ID文件：echo 1 > /data/zookeeper/myid

dataDir=/data/zookeeper

clientPort=2181

maxClientCnxns=0

admin.enableServer=true

admin.serverPort=8080

initLimit=5

syncLimit=2

server.1=192.168.199.140:2888:3888

server.2=192.168.199.141:2888:3888

server.3=192.168.199.142:2888:3888

192.168.199.141

创建集群ID文件：echo 2 > /data/zookeeper/myid

dataDir=/data/zookeeper

clientPort=2181

maxClientCnxns=0

admin.enableServer=true

admin.serverPort=8080

initLimit=5

syncLimit=2

server.1=192.168.199.140:2888:3888

server.2=192.168.199.141:2888:3888

server.3=192.168.199.142:2888:3888

192.168.199.142

创建集群ID文件：echo 3 > /data/zookeeper/myid

dataDir=/data/zookeeper

clientPort=2181

maxClientCnxns=0

admin.enableServer=true

admin.serverPort=8080

initLimit=5

syncLimit=2

server.1=192.168.199.140:2888:3888

server.2=192.168.199.141:2888:3888

server.3=192.168.199.142:2888:3888

配置释义：

initLimit：允许follower与leader连接和同步的时间，如果ZooKeeper管理的数据量很大，可以根据需要增加这个值。

syncLimit：允许follower与leader同步的时间，如果follower远远落后于leader，他们将被丢弃。

maxClientCnxns：限制单个客户端(由IP地址标识)对ZooKeeper集成中的单个成员的并发连接数(在套接字级别)。这是用来防止某些类型的DoS攻击，包括文件描述符耗尽。默认值是60。将此值设置为0完全删除了并发连接的限制。

admin.enableServer：是否开启管理服务

admin.serverPort=8080：管理服务开放端口

clientPort：监听客户端连接的端口

server.id=host:port:port 对于每台服务器，您首先需要指定一个仲裁端口，然后为ZooKeeper leader选举指定一个专用端口

各节点启动zookeeper：

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

集群管理命令：

url示例：/commands/[command name]

查看zookeeper集群服务状态：

http://192.168.199.142:8080/commands/stat

或

curl http://localhost:8080/commands/stat

由此查看到目前192.168.199.142这台是leader节点。

2、kafka集群配置(server.properties)

192.168.199.140

broker.id=0

listeners=PLAINTEXT://192.168.199.140:9092

advertised.listeners=PLAINTEXT://192.168.199.140:9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=2

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0

192.168.199.141

broker.id=1

listeners=PLAINTEXT://192.168.199.141:9092

advertised.listeners=PLAINTEXT://192.168.199.141:9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=2

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0

192.168.199.142

broker.id=2

listeners=PLAINTEXT://192.168.199.142:9092

advertised.listeners=PLAINTEXT://192.168.199.142:9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=2

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0

参数参考官网：

http://kafka.apache.org/documentation/#connect_configuring

各节点启动kafka：

bin/kafka-server-start.sh -daemon config/server.properties

创建topic：

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.199.140:9092

查看topic详细信息：

bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 192.168.199.140:9092

模拟生产者写消息：

$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.199.140:9092

This is my first event

This is my second event

模拟消费者读消息：

$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.199.140:9092

This is my first event

This is my second event

集群部署完成。