centos7 使用Kafka自带的zookeeper建立kafka集群(不要问我为什么!!)

前提

• 准备三台服务器或者三台虚拟机,使用虚拟机,强烈建议关闭防火墙,等配置完成后,再配置防火墙.
  192.168.1.32
  192.168.1.33
  192.168.1.34
• 在每一台虚拟机上安装好JDK. 下载好kafka的压缩包.

一

在每一台虚拟机上,创建kafka和zookeeper的文件和日志目录

mkdir -p /data/zookeeper/{data,log}
mkdir -p /data/kafka/{data,log}

二

第一台服务器写入

echo "1" >/data/zookeeper/data/myid

第二台服务器写入

echo "2" >/data/zookeeper/data/myid

第三台服务器写入

echo "3" >/data/zookeeper/data/myid

三.修改配置文件

• 把kafka压缩包解压, kafka不用安装, 解压即用.我的三个虚拟机的文件目录是一样的. 我的kafka的位置在
  /usr/local/install目录下. 进入kafka的conf目录. 首先修改zookeeper配置文件vi zookeeper.properties.

# 把配置好的zookeeper日志和文件,目录配置一下
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/log
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
#maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
tickTime=2000
initLimit=10
syncLimit=5
# 然后添加这三个
server.1=192.168.1.32:2888:3888
server.2=192.168.1.33:2888:3888
server.3=192.168.1.34:2888:3888

三台虚拟机的zookeeper的配置文件做同样的修改.

• 然后修改kafka的配置文件, vi server.properties, 以192.168.1.32为例, 为了方便看清, 删除了大部分注释.

############################# Server Basics #############################

# 这里的broker.id是唯一的, 三台虚拟机依次配置成1, 2, 3
broker.id=1

############################# Socket Server Settings #############################
# 这里的IP需要修改成当前虚拟机的IP
advertised.listeners=PLAINTEXT://192.168.1.32:9092

num.network.threads=3

num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################


# kafka的日志目录
log.dirs=/data/kafka/log
# 分区数.可以写成3个
num.partitions=3
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# 副本数改为3
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################
# zookeeper集群地址
zookeeper.connect=192.168.1.32:2181,192.168.1.33:2181,192.168.1.34:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################

group.initial.rebalance.delay.ms=0

四. 启动集群

PS: 一开始本来是使用独立的zookeeper, 不用kafka自带. 跑通后,但是就是这启动脚本实在是不会写. 而后转自带的zookeeper. 说多了都是泪. 这该死的知识储备

1. kafka依赖zookeeper, 启动kafka前必须先启动zookeeper
2. 把zookeeper和kafka做成服务, 便于开机自启动
3. 编写启动代码 zookeeper.service
   vi /lib/systemd/system/zookeeper.service
   写入:

[Unit]
Description=Zookeeper service
After=network.target
[Service]
Type=simple
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/install/jdk1.8.0_152/bin"
User=root
Group=root
ExecStart=/usr/local/install/kafka_2.12-2.7.0/bin/zookeeper-server-start.sh /usr/local/install/kafka_2.12-2.7.0/config/zookeeper.properties
ExecStop=/usr/local/install/kafka_2.12-2.7.0/bin/zookeeper-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target

---

kafka.service
vi /lib/systemd/system/kafka.service
写入:

[Unit]
Description=Apache Kafka server (broker1)
After=network.target  zookeeper.service
[Service]
Type=simple
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/install/jdk1.8.0_152/bin"
User=root
Group=root
ExecStart=/usr/local/install/kafka_2.12-2.7.0/bin/kafka-server-start.sh /usr/local/install/kafka_2.12-2.7.0/config/server.properties
ExecStop=/usr/local/install/kafka_2.12-2.7.0/bin/kafka-server-stop.sh
Restart=on-failure
[Install]
WantedBy=multi-user.target

三台虚拟机做同样的修改

4. 输入 systemctl daemon-reload
5. 设置集群开机自启动

systemctl enable zookeeper
systemctl enable kafka 

6. 启动zookeeper
   systemctl start zookeeper
   查看状态
   systemctl status zookeeper
   
7. 启动kafka
   systemctl start kafka
   查看kafka状态
   systemctl status kafka
   

五. 测试

在kafka解压目录/usr/local/install/kafka_2.12-2.7.0, 以启动一台虚拟机为生产者, 另外两台为消费者测试

1. 查看topic列表
   bin/kafka-topics.sh --list --zookeeper 192.168.1.34:2181
2. 创建topic
   bin/kafka-topics.sh --create -zookeeper 192.168.1.34:2181 -replication-factor 3 --partitions 3 --topic dcy-test
3. 在其中一台服务器创建生产者
   bin/kafka-console-producer.sh --broker-list 192.168.1.32:9092,192.168.1.33:9092,192.168.1.34:9092 --topic dcy-test
   
4. 另外两台服务器创建消费者
   bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.32:9092,192.168.1.33:9092,192.168.1.34:9092 --topic dcy-test --from-beginning
   
   测试通过!!
   提醒: 如果 使用同一类型云服务器(比如阿里云), 请把zookeeper的IP更改为内网IP, 这样更合理