kafka常用指令总结

kafka指令

1.常用指令

1.1 Adding and removing topics

add：

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
       --partitions 20 --replication-factor 3 --config x=y

remove:

1.2 Modifying topics

add partition：

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name
       --partitions 40

add configes:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --config x=y

To remove a config:

> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name --deleteConfig x

delete config (不使用):

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name

1.3 balancing leadership

 bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot

前提是需要加上配置

 auto.leader.rebalance.enable=true

1.4 checking consumer position

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect localhost:2181 --group test
Group           Topic                          Pid Offset          logSize         Lag             Owner
my-group        my-topic                       0   0               0               0               test_jkreps-mn-1394154511599-60744496-0
my-group        my-topic                       1   0               0               0               test_jkreps-mn-1394154521217-1a0be913-0

1.5 Expanding your cluster(水平扩展集群)

• –generate: 在此模式下，给定主题列表和代理列表，该工具会生成候选重新分配，以将指定主题的所有分区移动到新代理。此选项仅提供了一种方便的方法，可在给定主题和目标代理列表的情况下生成分区重新分配计划。
• –execute:在此模式下，该工具将根据用户提供的重新分配计划启动分区的重新分配。 （使用–reassignment-json-file选项）。这可以是由管理员手工制作的自定义重新分配计划，也可以使用–generate选项提供
• –verify: 在此模式下，该工具将验证最后一次–execute期间列出的所有分区的重新分配状态。状态可以是成功完成，失败或正在进行中

1.由于该工具接受主题的输入列表作为json文件，因此首先需要确定要移动的主题并创建json文件，如下所示 -

> cat topics-to-move.json
{"topics": [{"topic": "foo1"},
            {"topic": "foo2"}],
 "version":1
}

2.一旦json文件准备就绪，使用分区重新分配工具生成候选分配 -

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate
Current partition replica assignment

{"version":1,
 "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
               {"topic":"foo1","partition":0,"replicas":[3,4]},
               {"topic":"foo2","partition":2,"replicas":[1,2]},
               {"topic":"foo2","partition":0,"replicas":[3,4]},
               {"topic":"foo1","partition":1,"replicas":[2,3]},
               {"topic":"foo2","partition":1,"replicas":[2,3]}]
}

Proposed partition reassignment configuration

{"version":1,
 "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
               {"topic":"foo1","partition":0,"replicas":[5,6]},
               {"topic":"foo2","partition":2,"replicas":[5,6]},
               {"topic":"foo2","partition":0,"replicas":[5,6]},
               {"topic":"foo1","partition":1,"replicas":[5,6]},
               {"topic":"foo2","partition":1,"replicas":[5,6]}]
}

3.该工具生成一个候选分配，将所有分区从主题foo1，foo2移动到代理5,6。 但请注意，此时分区移动尚未开始，它只是告诉您当前的分配和建议的新分配。 应保存当前分配，以防您想要回滚它。 新的分配应保存在json文件中（例如expand-cluster-reassignment.json），以便使用–execute选项输入工具，如下所示 -

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
Current partition replica assignment

{"version":1,
 "partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
               {"topic":"foo1","partition":0,"replicas":[3,4]},
               {"topic":"foo2","partition":2,"replicas":[1,2]},
               {"topic":"foo2","partition":0,"replicas":[3,4]},
               {"topic":"foo1","partition":1,"replicas":[2,3]},
               {"topic":"foo2","partition":1,"replicas":[2,3]}]
}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
 "partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
               {"topic":"foo1","partition":0,"replicas":[5,6]},
               {"topic":"foo2","partition":2,"replicas":[5,6]},
               {"topic":"foo2","partition":0,"replicas":[5,6]},
               {"topic":"foo1","partition":1,"replicas":[5,6]},
               {"topic":"foo2","partition":1,"replicas":[5,6]}]
}

4.检查状态

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition [foo1,0] completed successfully
Reassignment of partition [foo1,1] is in progress
Reassignment of partition [foo1,2] is in progress
Reassignment of partition [foo2,0] completed successfully
Reassignment of partition [foo2,1] completed successfully
Reassignment of partition [foo2,2] completed successfully

1.6 自定义分区分配和迁移

1.创建分区文件 custom-reassignment.json

cat custom-reassignment.json
{
	"version": 1,
	"partitions": [{
		"topic": "foo1",
		"partition": 0,
		"replicas": [5, 6]
	}, {
		"topic": "foo2",
		"partition": 1,
		"replicas": [2, 3]
	}]
}

2.使用json文件执行分区操作

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment

{"version":1,
 "partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},
               {"topic":"foo2","partition":1,"replicas":[3,4]}]
}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
 "partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
               {"topic":"foo2","partition":1,"replicas":[2,3]}]
}

3.确认状态

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment

{"version":1,
 "partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},
               {"topic":"foo2","partition":1,"replicas":[3,4]}]
}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
 "partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
               {"topic":"foo2","partition":1,"replicas":[2,3]}]
}

1.7 Increasing replication factor(新增副本数)

逻辑和上两个一致 通过json文件 手动修改副本数据位置

提供json文件:

> cat increase-replication-factor.json
{
	"version": 1,
	"partitions": [{
		"topic": "foo",
		"partition": 0,
		"replicas": [5, 6, 7]
	}]
}

1.8 启动 消费 等指令

1.启动zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties &

2.启动kafka

bin/kafka-server-start.sh config/server.properties &

3.停止kafka

bin/kafka-server-stop.sh

4.停止zookeeper

bin/zookeeper-server-stop.sh

5.创建topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

6.展示topic

bin/kafka-topics.sh --list --zookeeper localhost:2181

7.描述topic

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

8.生产者：

bin/kafka-console-producer.sh --broker-list 130.51.23.95:9092 --topic my-replicated-topic

9.消费者：

bin/kafka-console-consumer.sh --zookeeper 130.51.23.95:2181 --topic test --from-beginnin