kafka3.8.0笔记（Centos7）

孤独天狼

已于 2025-01-09 09:55:45 修改

阅读量421

点赞数 2

文章标签：笔记

于 2024-10-30 17:33:09 首次发布

本文链接：https://blog.youkuaiyun.com/u013226533/article/details/143370663

版权

基础信息

1、需要jdk8以上

2、官网地址：Apache Kafka

3、安装包下载地址：https://archive.apache.org/dist/kafka/3.8.0/kafka_2.13-3.8.0.tgz

4、zookeeper下载地址：https://dlcdn.apache.org/zookeeper/zookeeper-3.9.3/apache-zookeeper-3.9.3-bin.tar.gz

5、EFAK（集群监控工具）地址：EFAK

6、EFAK博客：哥不是小萝莉 - 博客园

7、特别感谢尚硅谷老师的视频

地址：45-kafka-消费数据-事务数据的隔离级别_哔哩哔哩_bilibili

安装

tar -xvf kafka_2.13-3.8.0.tgz -C /usr/local/                        #解压
tar -zxvf apache-zookeeper-3.9.3-bin.tar.gz -C /usr/local/          #解压

#配置zookeeper
cd /usr/local/apache-zookeeper-3.9.3-bin/conf
cp zoo_sample.cfg zoo.cfg

#启动zookeeper
cd /usr/local/apache-zookeeper-3.9.3-bin/bin
./zkServer.sh start
    admin.serverPort=端口号                        #在文件末尾添加，主要是解决占用8080端口问题
ps -ef|grep zookeeper                             #查看是否启动 
netstat -nlpt                                     #查看端口  

#配置kafka
cd /usr/local/kafka_2.13-3.8.0/config
vim server.properties
 #在Socket Server Settings部分添加以下内容
    listeners=PLAINTEXT://0.0.0.0:9092                    
    advertised.listeners=PLAINTEXT:/IP地址:9092
 #在Zookeeper部分修改Zookeeper的连接地址
    zookeeper.connect=zookeeper的IP地址:2181



#启动kafka  （方式一）

vim /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)

[Service]
Type=simple
Environment="JAVA_HOME=/etc/jdk-23.0.1"
User=root
Group=root
ExecStart=/opt/package/kafka_2.13-3.9.0/bin/kafka-server-start.sh /opt/package/kafka_2.13-3.9.0/config/server.properties
ExecStop=/opt/package/kafka_2.13-3.9.0/bin/kafka-server-stop.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

#加载配置文件
systemctl daemon-reload
#启动kafka
systemctl start kafka.service
#设置开机自启动
systemctl enable kafka.service
#查看启动状态
systemctl status kafka.service 


#启动kafka  （方式二）
cd /usr/local/kafka_2.13-3.8.0/bin
./kafka-server-start.sh ../config/server.properties &   
ps -ef|grep kafka                                 #查看是否启动 
netstat -nlpt                                     #查看端口  

#########################docker安装kafka########################################
docker pull apache/kafka3.7.0                                     #下载镜像
docker run -p 9092:9092  apache/kafka3.7.0                        #启动镜像
docker exec -it 容器ID /bin/bash                                   #进入容器命令行
cd /etc/kafka/docker                                              #进入kafka路径，查看配置文件
exit                                                              #退出容器命令行
makdir /opt/kafka/docker                                          #创建文件夹，存放配置文件
docker cp 容器ID:/etc/kafka/docker/server.properties ./           #将容器内的文件复制到宿主机
vim server.properties
 #在Socket Server Settings部分添加以下内容
    listeners=PLAINTEXT://0.0.0.0:9092                    
    advertised.listeners=PLAINTEXT:/IP地址:9092
 #在Zookeeper部分修改Zookeeper的连接地址
    zookeeper.connect=zookeeper的IP地址:2181
docker run --volume /opt/kafka/docker:/mut/shared/config -p 9092:9092 apache/kafka3.7.0    #将本地文件映射到容器内部    /opt/kafka/docker是宿主机路径；  /mut/shared/config是固定写法

#在idea安装插件kafka进行连接
搜索 kafka  进行安装


############################EFAK 集群监控工具安装####################################
tar -zxvf kafka-eagle-bin-3.0.1.tar.gz                 #解压安装包
cd kafka-eagle-bin-3.0.1/                              #进入解压目录内
tar -zxvf efak-web-3.0.1-bin.tar.gz -C /usr/local/     #解压安装包

连接mysql
create database if not exists ke ;                      #创建数据库  
   
vim system-config.properties                            #修改配置文件
  #修改zookeeper的配置信息
    efak.zk.cluster.alias=cluster1,cluster2             #修改为   cluster1
    cluster1.zk.list=tdn1:2181,tdn2:2181,tdn3:2181      #修改为   IP地址:2181
    cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181   #该行注释掉
  #修改mysql的配置信息
    efak.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
    efak.username=root
    efak.password=123456

vim /etc/profile                                         #配置环境变量
    #在文件末尾添加以下内容
        export KE_HOME=/usr/local/efak-web-3.0.1        
        export PATH=$KE_HOME/bin:$PATH
source /etc/profile                                      #重新加载配置文件  
cd /usr/local/efak-web-3.0.1/bin
./ke.sh start                                            #启动

#出现以下内容代表启动成功
Welcome to
    ______    ______    ___     __ __
   / ____/   / ____/   /   |   / //_/
  / __/     / /_      / /| |  / ,<   
 / /___    / __/     / ___ | / /| |  
/_____/   /_/       /_/  |_|/_/ |_|  
( Eagle For Apache Kafka® )

Version v3.0.1 -- Copyright 2016-2022
*******************************************************************
* EFAK Service has started success.
* Welcome, Now you can visit 'http://192.168.111.132:8048'
* Account:admin ,Password:123456
*******************************************************************
* <Usage> ke.sh [start|status|stop|restart|stats] </Usage>
* <Usage> https://www.kafka-eagle.org/ </Usage>
*******************************************************************

kakfa概念

架构图

1、基础名词

Producer：生产者

Consumer：消费者

Topic：主题，每个Topic可以有一个或多个partition（分区），如不指定分区默认就是1个分区

Partition：分区

生产者Offset（偏移量），是标识每个分区中消息的唯一位置，从0开始，有序的

1.生产者发送一条消息到kafka的broker（服务器）的某个topic（主题）下某个partition（分区）中

2.kafka内部会为每条消息分配一个唯一的offset，该offset就是该消息在partition中的位置

消费者Offset（偏移量）

每个消费组启动就开始监听消息，默认从消息的最新位置开始监听

情况一： 分区中还没有发送过消息，则最新的位置是0

情况二：分区中已发送过消息，则最新的位置就是生产者offset的下一个位置

消费者消费消息后，如果不提交确认（ack），则offset不更新，提交了才更新

2、Replica副本

Replica（副本）：保证集群某个节点故障时，该节点的Partition（分区）数据不会丢失，一个Topic（主题）的每个Partition（分区）可以有1个或多个副本，其中Replica（副本）又分为Leader Replica（主副本）:用于读写和Follower Replica（从副本）：用于备份

3、生产者生产消息的分区策略

含义：生产者根据不同的策略往topic（主题）中写数据

3.1、默认策略：使用的是：BuiltlnPartitioner类

1.1 有key时，对key进行hash并向上取整取余（%）分区数

1.2 没key时，随机数取余（%）分区数

3.2、轮询策略：使用的是：RoundRobinPartitioner类（接口是：partitioner）

注意：通过源码跟踪测试发现并不是真正的轮询。

3.3、自定义策略：实现Partitioner接口

4、消费者消费消息的分区策略

4.1 默认策略：RangeAssignor（范围分配）：根据消费者组内的消费者数量和主题的分区数量，均匀的为每个消费者分配分区

4.1.1 计算消费者应得的分区数：分区总数 / 消费者数 =消费者应得的分区数，如有除不尽，有余数时，将把余数分给第一个消费者；

4.1.2 具体分配：按分区编号，从0开始为消费者分配分区数

样例：比如有10个分区，3个组，那么组1应得到分区0，1，2，3 ；组2应得4，5，6；组3应得7，8，9

4.2 RoundRobinAssignor（轮询分配）

4.3 StickyAssignor（粘性分配）：尽量保持现有的分区分配不变，仅对新加入的或离开的消费者进行分区调整，这样能保证只做少量的调整，所以叫 “粘性”分配

4.4 CooperativeStickyAssignor（协作粘性分配）：增加对协作式重新平衡的支持，即消费者可以在离开消费组之前通知协调器，以便协调器可以预先计划分区迁移，而不是消费者突然离开时在进行分区重分配

5、主题创建流程

6、生产者发送消息流程

7、消费者拉取数据流程

8、幂等性

作用：是为了解决数据重复以及乱序的问题

原理：1.在ProducerState中保存生成者的生产状态，里面有5（固定值）条数据

1.1 比对当前数据会与状态里的数据是否相同；

1.1.1 相同：使用状态里的数据；

1.1.2 不相同：告知Producer将有问题的数据重新发送；

1.2 比对当前数据会与状态里的数据是否连续；

1.2.1 连续：从队列的头部开始删除，增加新数据，确保队列只能有5条数据

1.2.1 不连续：告知Producer将有问题的数据重新发送；

2.幂等性只能保证一个分区内的数据有序和不重复；

3.幂等性使用生产者ID+顺序号保证数据的有序和不重复；

注意：生产者只要一重启，生产者ID就会改变，也就是幂等性会失效

9、事务

作用：解决同一个分区内的幂等性失效问题

10、事件（消息、数据）的存储

1 kakfa的所有事件（消息、数据）存储在 log.dirs=/tmp/kafka-logs里（配置文件）

2 kakfa的所有事件（消息、数据）都已日志文件的方式保存；

3 为了避免日志文件过大，日志文件被存放在多个日志目录下，日志目录的命名规则：主题名 - 分区ID

4 每次消费一个消息并提交以后，会保存当前消费到最近的一个offset

5 在kakfa中，有一个__consumer_offset的topic，消费者消费提交的offset信息会写入到该topic中，__consumer_offset保存了每个consumer group某一时刻提交的offset信息，__consumer_offset默认有50个分区

6 每个目录会包含以下文件

xxxxxx.index 消息索引文件

xxxxxx.log 消息数据文件

xxxxxx.timeindex 消息的时间戳索引文件

xxxxxx.snapshot 快照文件，生产者发生故障或重启时快速恢复并继续之前的操作

leader-epoch-checkpoint 记录每个分区当前领导者的epoch以及领导者开始写入消息时的起始偏移量

partition.metadata 存储关于特定分区的元数据（metadata）信息

存储流程

kafka性能评估

概要

1、存储的一条数据大小在10K左右是最优的
2、kafka只接收字节数组
3、kafka的通信是用TCP协议
4、kafka的分区数最好不要超过10个

估算kafka的参考项
磁盘：影响最大的是生产者的读写性能，选用HDD机械（可设置多分区、多目录），计算磁盘大小要，数据有1G，在加上副本数量
内存：影响最大的是消费者，因为消费者是从内存中获取数据，当存储容量不够时，会从磁盘拉取数据
网络：影响生产者和消费者，因为写和读会占用网络
CPU：主要是用来做数据压缩，对kafka来说不是瓶颈

常用命令

cd /usr/local/kafka_2.13-3.8.0/bin   
#启动kafka
./kafka-server-start.sh ../config/server.properties &
#查看事件（消息、数据）存储位置
./kafka-storage.sh info -c ../config/server.properties
#生成集群ID
./kafka-storage.sh random-uuid

############################### topic的操作 ########################################
#创建topic的脚本  不写参数是查看命令帮助
./kafka-topics.sh
#创建主题
./kafka-topics.sh --create --topic test-topic --bootstrap-server 127.0.0.1:9092
#查看主题
./kafka-topics.sh --list  --bootstrap-server 127.0.0.1:9092
#查看主题详细信息
./kafka-topics.sh --describe --topic test-topic --bootstrap-server 127.0.0.1:9092
#删除主题
./kafka-topics.sh --delete --topic test-topic --bootstrap-server 127.0.0.1:9092
#修改主题分区
./kafka-topics.sh --alter --topic test-topic --partitions 5 --bootstrap-server 127.0.0.1:9092
#创建主题时，设置分区数和副本数（副本数必须与集群节点数一致，否则报错）
./kafka-topics.sh --create test-topic --partitions 5 --replication-factor 2 --bootstrap-server 127.0.0.1:9092

############################### 给主题写事件操作 ########################################
#写事件的脚本  不写参数是查看命令帮助
./kafka-console-producer.sh
#写事件
./kafka-console-producer.sh --topic test-topic --bootstrap-server 127.0.0.1:9092

#消费事件的脚本  不写参数是查看命令帮助
./kafka-console-consumer.sh
#消费事件
./kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server 127.0.0.1:9092

################################ 组的操作##################################
#查看消费组的消费信息
./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group myGropu10  --describe

#手动重置偏移量到最早的下一个位置
./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group myGropu10 --topic test-topic --reset-offsets --to-earliest --execute

#手动重置偏移量到最后一条的下一个位置
./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group myGropu10 --topic test-topic --reset-offsets --to-latest --execute

配置文件

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#
# This configuration file is intended for use in ZK-based mode, where Apache ZooKeeper is required.
# See kafka.server.KafkaConfig for additional details and defaults
#

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. If not configured, the host name will be equal to the value of
# java.net.InetAddress.getCanonicalHostName(), with PLAINTEXT listener name, and port 9092.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=PLAINTEXT://0.0.0.0:9092

# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
#advertised.listeners=PLAINTEXT://your.host.name:9092
advertised.listeners=PLAINTEXT://192.168.111.132:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
# 当满足条数时，就写入磁盘
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
# 生成文件的大小
#log.segment.bytes=1073741824
#滚动生成的时间
#log.roll.ms =5
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000


############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

SpringBoot3整合Kakfa

//在pom.xml中添加start依赖 
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
 </dependency>

//在application.yml中添加

#kafka的配置信息
spring:
  kafka:
    bootstrap-servers: IP地址:9092