Kafka1.0日常维护

最新推荐文章于 2025-07-01 14:48:59 发布
置顶 最新推荐文章于 2025-07-01 14:48:59 发布
阅读量1k 收藏 3
点赞数
CC 4.0 BY-SA版权
分类专栏： Kafka 文章标签： 大数据 bigdata kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/GPF19950411/article/details/79430237

日常维护操作

4.1 创建topic两个备份6个分区：

./kafka-topics.sh --create --zookeeper app212:2181 --replication-factor 2 --partitions 6 --topic t_test

4.2 查看存在的topic：

./kafka-topics.sh --zookeeper app212:2181 --list

4.3 查看topic详细信息：


./kafka-topics.sh --describe --zookeeper app212:2181 --topic t_test

topic详细信息

4.4 生产数据：

./kafka-console-producer.sh --broker-list localhost:9092 --topic t_test

模拟生产数据

4.5 消费数据：

消费数据：
./kafka-console-consumer.sh --zookeeper app211:2181 --topic t_test

模拟消费数据

4.6 查看topic某分区偏移量最大（小）值

./kafka-run-class.sh kafka.tools.GetOffsetShell --topic t_test  --time -1 --broker-list 10.242.167.216:9092,10.242.167.217:9092,10.242.167.218:9092 --partitions 0
time为-1时表示最大值，time为-2时表示最小值

偏移量

4.7 删除topic

只会删除zookeeper中的元数据，消息文件须手动删除

./kafka-topics.sh --zookeeper app212:2181 --delete --topic t_test

4.8 增加topic分区数（只能增加）

./kafka-topics.sh --zookeeper app212:2181  --alter --topic t_test --partitions 10

4.8.1 查看group消费情况

kafka-consumer-groups.sh --bootstrap-server host:9092 --group group_name --describe

4.9 增加topic副本数

创建partitions-extension-push-token-topic.json
vim /opt/kafka/partitions-extension-push-token-topic.json

{
    "partitions": [
        {
            "topic": "t_test",
            "partition": 0,
            "replicas": [
                4,
                5,
				6
            ]
        },
        {
            "topic": "t_test",
            "partition": 1,
            "replicas": [
                5,
                6,
				1
            ]
        },
        {
            "topic": "t_test",
            "partition": 2,
            "replicas": [
                6,
                1,
				2
            ]
        },
		{
            "topic": "t_test",
            "partition": 3,
            "replicas": [
                1,
                2,
				3
            ]
        },
		{
            "topic": "t_test",
            "partition": 4,
            "replicas": [
                2,
                3,
				4
            ]
        },
		{
            "topic": "t_test",
            "partition": 5,
            "replicas": [
                3,
                4,
				5
            ]
        }
    ],
    "version": 1
}

执行变更：

命令：./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[4,5],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

验证结果：

./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --verify

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --verify
Status of partition reassignment: 
Reassignment of partition t_test-5 completed successfully
Reassignment of partition t_test-2 completed successfully
Reassignment of partition t_test-0 completed successfully
Reassignment of partition t_test-4 completed successfully
Reassignment of partition t_test-1 completed successfully
Reassignment of partition t_test-3 completed successfully

查看结果
变更前：

[root@dbmspreapp212 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic t_test
Topic:t_test	PartitionCount:6	ReplicationFactor:2	Configs:
	Topic: t_test	Partition: 0	Leader: 4	Replicas: 4,5	Isr: 4,5
	Topic: t_test	Partition: 1	Leader: 5	Replicas: 5,6	Isr: 5,6
	Topic: t_test	Partition: 2	Leader: 6	Replicas: 6,1	Isr: 6,1
	Topic: t_test	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: t_test	Partition: 4	Leader: 2	Replicas: 2,3	Isr: 2,3
	Topic: t_test	Partition: 5	Leader: 3	Replicas: 3,4	Isr: 3,4

变更后

[root@dbmspreapp212 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic t_test
Topic:t_test	PartitionCount:6	ReplicationFactor:3	Configs:
	Topic: t_test	Partition: 0	Leader: 4	Replicas: 4,5,6	Isr: 4,5,6
	Topic: t_test	Partition: 1	Leader: 5	Replicas: 5,6,1	Isr: 5,6,1
	Topic: t_test	Partition: 2	Leader: 6	Replicas: 6,1,2	Isr: 6,1,2
	Topic: t_test	Partition: 3	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: t_test	Partition: 4	Leader: 2	Replicas: 2,3,4	Isr: 2,3,4
	Topic: t_test	Partition: 5	Leader: 3	Replicas: 3,4,5	Isr: 3,4,5

4.10 减少topic副本数

类似于增加副本数，只需要更改json文件的replicas配置项

4.11 Kafka集群扩容（重分配topic分区）

创建topic-to-move.json
vim /opt/kafka/topic-to-move.json

{
    "topics": [
        {
            "topic": "t_test"
        }
    ],
    "version": 1
}

生成迁移分配规则json文件:

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file ../topic-to-move.json --broker-list "1,2,3,4,5,6,8" --generate

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file ../topic-to-move.json --broker-list "1,2,3,4,5,6,8" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[5,6],"log_dirs":["any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[1,8],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[5,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[6,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[4,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[8,6],"log_dirs":["any","any"]}]}

其中的Current partition replica assignment指的是迁移前的partition replica；Proposed partition reassignment configuration 指的就是迁移分配规则json。需要将该json文件保存到json文件中(如expand-cluster-reassignment.json)
vim /opt/kafka/expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[1,8],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[5,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[6,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[4,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[8,6],"log_dirs":["any","any"]}]}

执行迁移分配

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --execute

[root@dbmspreapp212 bin]# kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[5,6],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

验证

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --verify

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --verify
Status of partition reassignment: 
Reassignment of partition t_test-5 completed successfully
Reassignment of partition t_test-2 completed successfully
Reassignment of partition t_test-0 completed successfully
Reassignment of partition t_test-4 completed successfully
Reassignment of partition t_test-6 completed successfully
Reassignment of partition t_test-1 completed successfully
Reassignment of partition t_test-3 completed successfully

结果：

[root@dbmspreapp212 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic t_test
Topic:t_test	PartitionCount:7	ReplicationFactor:2	Configs:
	Topic: t_test	Partition: 0	Leader: 8	Replicas: 8,6	Isr: 6,8
	Topic: t_test	Partition: 1	Leader: 1	Replicas: 1,8	Isr: 1,8
	Topic: t_test	Partition: 2	Leader: 1	Replicas: 2,1	Isr: 1,2
	Topic: t_test	Partition: 3	Leader: 2	Replicas: 3,2	Isr: 2,3
	Topic: t_test	Partition: 4	Leader: 3	Replicas: 4,3	Isr: 3,4
	Topic: t_test	Partition: 5	Leader: 4	Replicas: 5,4	Isr: 4,5
	Topic: t_test	Partition: 6	Leader: 5	Replicas: 6,5	Isr: 5,6

4.12 Kafka集群节点下线

和扩容相反，--broker-list "1,2,3,4,5,8",比如下线节点6，然后生成文件等进行操作。
重新leader、重新均衡,下线节点完成后重新均衡下
./kafka-preferred-replica-election.sh --zookeeper localhost:2181

【大数据】Flink + Kafka 实现通用流式数据处理详解
congge
10-27 1万+
Flink + Kafka 实现通用流式数据处理详解
ELK+Filebeat+Kafka+Zookeeper日志分析系统搭建
weixin_67510296的博客
07-17 1412
随着业务量的增长，每天业务服务器将会产生上亿条的日志，单个日志文件达几个GB，这时我们发现用Linux自带工具，cat grep awk 分析越来越力不从心了，而且除了服务器日志，还有程序报错日志，分布在不同的服务器，查阅繁琐。 导致 大量不同种类的日志成为了运维人员的负担，不方便管理。 单个日志文件巨大，无法使用常用的文本工具分析，检索困难。 日志分布在多台不同的服务器上，业务一旦出现故障，需要一台台查看日志。 ...
Kafka常见运维命令.md
05-31
Kafka常见运维命令文档，Kafka常见运维命令文档，Kafka常见运维命令文档
kafka集群维护
weixin_34363171的博客
09-14 152
kafka集群基本信息实时查看和修改： 集群信息实时查看（topic工具) kafka-topics,sh --list --zookeeper x.x.x.x kafka-topics.sh --describe --zookeeper x.x.x.x --topic topic_name 集群信息实时修改(topic工具） 提高...
Kafka日常运维命令总结
huanhuanzhou blog
07-01 997
【代码】Kafka日常运维命令总结。
kafka日常运维工作
weixin_43046724的博客
02-26 1994
一、创建/删除/重建topic 1、创建：./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 10 --topicTOPIC_NAME 2、删除：./bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topicTOPIC_NAME 3、查看状态和分区负载详情：./bin/kafka-topics.s..
kafka常用运维命令
shengpli′s blog
01-15 1008
一、broker 在每一台节点上启动broker nohup bin/kafka-server-start.sh config/server.properties & 停止 bin/kafka-server-stop.sh config/server.properties 二、topic 创建topic bin/kafka-topics.sh –create –zookeeper
kafka知识体系-日常运维命令
weixin_30729609的博客
07-04 250
本文主要讲解kafka日常运维的命令，包括topic管理、性能测试脚本。 kafka版本0.10.0，安装步骤见大数据平台搭建-kafka集群的搭建 常用脚本 如下所有的命令均基于KAFKA_HOME=/wls/oracle/kafka ,服务器列表如下： 10.20.112.59 10.20.112.64 10.20.112.65 10.20.116.129 10.20.116.175 创建to...
一文读懂 kafka 的事务机制
明哥的IT随笔
09-16 2273
1 前言 大家好，我是明哥！ KAFKA 作为开源分布式事件流平台，在大数据和微服务领域都有着广泛的应用场景，是实时流处理场景下消息队列事实上的标准。用一句话概括，KAFKA 是实时数仓的基石，是事件驱动架构的灵魂。 但是一些技术小伙伴，尤其是一些很早就开始使用 KAFKA 的技术小伙伴们，对 KAFKA 的发展趋势和一些新特性，并不太熟悉，在使用过程中也踩了不少坑。 有鉴于此，我们会通过一系列 KAFKA 相关博文，专门讲述 KAFKA 的这些新特性。 本文是该系列文章之一，讲述 KAFAK 的事务机制。
看完这篇Kafka，你也许就会了Kafka
AaronCao
01-13 19万+
Kafka学习 文章目录Kafka学习1. Kafka简介1.1 Kafka消费模式1.2 Kafka的基础架构1.3 Kafka的安装和使用2. Kafka高级2.1 工作流程2.2 文件存储2.3 生产者分区策略2.4 生产者ISR2.4.1 副本数据同步策略2.4.2 ISR(同步副本集)2.5 生产者ack机制2.6 数据一致性问题2.7 ExactlyOnce3. 消费者分区分配策略3.1 分区分配策略3.2 消费者offset的存储3.3 消费者组案例4. 高效读写&Zookeeper作
大厂消息中间件高频面试题与知识点整理（RabbitMQ、RocketMQ、ActiveMQ、Kafka、ZeroMQ、MetaMQ）
热门推荐
张彦峰的博客
04-07 172万+
互联网高频面试之消息中间件基本问题和知识点整理分析
Kafka_2.11-1.1.0
05-06
Kafka_2.11-1.1.0
kafka使用手册
02-01
kafka入门与使用的详细教程，你值得拥有。
kafka集群部署文档(部署，运维，FAQ)
06-27
kafka集群部署文档及kafka详解，包括kafka常用命令，详细部署说明，如何运维以及一些FAQ，下载app注册免费获取：http://m3w.cn/jcsh
kafka日常操作
weixin_39956784的博客
07-14 240
创建topic /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic zx037 查看topic详细情况 /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic ots_to_shenzheng #返回结果 Topic:ots_to_she
Kafka维护问题总结
杨春建的博客
10-07 2067
原文地址：https://my.oschina.net/u/1169607/blog/353730 1 Kafka操作日志的清理方法 Kafka0.8版本长时间运行过程中，在kafka_2.8.0-0.8.0/logs目录下产生了大量的kafka-request.log.*和server.log.*文件，其中*代表日期和时间，比如kafka-request.log.2014-12-08
Kafka常用运维命令
东方小硕
12-06 490
使用 权限管理 /opt/bigdata/app/kafka/bin/kafka-configs.sh --zookeeper zkurl --entity-type topics --entity-name yyj1 --alter --add-config join.isr.wait.seconds=600 # 关于broker间的通信，需要一个admin用户 bin/kafka-configs.sh --zookeeper zkurl --alter --add-config 'SCRAM-SHA-
Kafka 的日常基本操作命令汇总
天道酬勤
04-23 450
集群环境是 CDH 环境 5.15.0 broker 有三台 192.168.17.26 192.168.17.27 192.168.17.28 zk 有三台 192.168.17.21 192.168.17.22 192.168.17.23 生产者shell kafka-console-producer --broker-list 192.168.17.26:9092,192.168.17...
15、kafka集群维护
lixiang987654321的专栏
08-16 394
kafka集群维护1、集群信息实时查看工具（topic工具）2、kafka集群leader平衡机制3、集群分区日志迁移 1、集群信息实时查看工具（topic工具） （1）列出集群当前所有可用的topic kafka-topics.sh --list --zookeeper kafka1:2181 （2）查看集群特定的topic信息 /bin/kafka-topics.sh --describe ...
java 一小时写百万次kafka数据，数据 按数据中时间小时和id写入不同文件
最新发布
07-24
同时，要注意文件按小时和ID分开，例如：`2022050112_ID001.txt`，其中2022050112表示2022年5月1日12时，ID001是具体的ID。 另外，消费者需要定期关闭已经完成的小时的文件（例如，当前时间已经进入下一个小时，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值