版本 kafka_2.10-0.8.2.1
初步了解一下一下kafka的基本用法。
win7 64 bit
jdk1.7
1)下载kafka_2.10-0.8.2.1.tgz并解压（整合scala）
2)修改配置文件(可选)
config/zookeeper.properties
dataDir=/zookeeper
这么写建到d盘根目录
dataDir=zookeeper
这么写建到bin/windows目录
3)启动zookeeper（窗口1)
cd bin\windows
zookeeper-server-start ..\..\config\zookeeper.properties
或
zookeeper-server-start ../../config/zookeeper.properties
4)zookeeper测试（窗口2)
echo stat | nc 192.168.25.42 2181
D:\fk\tech\kafka\kafka_2.10-0.8.2.1\kafka_2.10-0.8.2.1\bin\windows>zookeeper-shell.bat 192.168.25.42:2181
注： 这里没有提示符
D:\fk\tech\cloud\zookeeper\zookeeper-3.4.6\bin>zkcli.cmd 192.168.25.42:2181Connecting to localhost:2181
ls /测试ok
5)启动kafka （窗口3)
kafka-server-start.bat ..\..\config\server.properties
6)创建topic（窗口4)
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic mytopic
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - topics .
bat -- create -- zookeeper localhost : 2181 -- replication - factor 1 -- partitions 3 --
topic mytopic
Created topic "mytopic" .
7)查看topic（窗口4)
kafka-topics.bat --describe --zookeeper localhost:2181 --topic mytopic
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - topics
bat -- describe -- zookeeper localhost : 2181 -- topic mytopic
Topic : mytopic PartitionCount : 3 ReplicationFactor : 1 Configs :
Topic : mytopic Partition : 0 Leader : 0 Replicas : 0 Isr : 0
Topic : mytopic Partition : 1 Leader : 0 Replicas : 0 Isr : 0
Topic : mytopic Partition : 2 Leader : 0 Replicas : 0 Isr : 0
8）发布/生产消息窗口（窗口5)
kafka-console-producer.bat --broker-list localhost:9092 --topic mytopic
有一个错误提示，但是不影响(kafa的bug KAFKA-1711，KAFKA-1824,预计 0.9修复 )
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - consol
- producer . bat -- broker - list localhost : 9092 -- topic mytopic
[ 2015 - 09 - 30 10 : 06 : 11 , 420 ] WARN Property topic is not valid ( kafka . utils . Verifia
leProperties )
9）消费消息窗口 （窗口6)
kafka-console-consumer.bat --zookeeper localhost:2181 --from-beginning --topic mytopic
10）producer窗口输入，可以在consumer窗口输出
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - console
- producer . bat -- broker - list localhost : 9092 -- topic mytopic
[ 2015 - 09 - 30 10 : 06 : 11 , 420 ] WARN Property topic is not valid ( kafka . utils . Verifiab
leProperties )
hello
kafka
error
this is a test msg from producer
中文
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - console
- consumer . bat -- zookeeper localhost : 2181 -- from - beginning -- topic mytopic
hello
kafka
error
this is a test msg from producer
中文
10)znode查看
ls /
[ consumers , config , controller , admin , brokers , zookeeper , controller_epoch
ls / consumers
[ console - consumer - 21728 ]
ls / config
[ topics , changes ]
ls / controller
[]
ls / admin
[ delete_topics ]
ls / brokers
[ topics , ids ]
ls / zookeeper
[ quota ]
ls / controller_epoch
[]
11)检查offset(窗口4)
注意：group为consumerid,需要在zk里面查ls /consumer
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - consume
r - offset - checker . bat -- topic mytopic -- zookeeper localhost : 2181 -- broker - info 1
27.0 . 0.1 : 9092 -- group console - consumer - 64061
Group Topic Pid Offset logSize
Lag Owner
console - consumer - 64061 mytopic 0 4 4
0 console - consumer - 64061 _zynb0411 - 1443580144073 - 95285fc3
- 0
console - consumer - 64061 mytopic 1 1 1
0 console - consumer - 64061 _zynb0411 - 1443580144073 - 95285fc3
- 0
console - consumer - 64061 mytopic 2 1 1
0 console - consumer - 64061 _zynb0411 - 1443580144073 - 95285fc3
- 0
BROKER INFO
0 -> zynb0411 : 9092
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows >
12)simple-consumer(底层操作)
指定offset
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - simple -
consumer - shell . bat -- topic mytopic -- offset 1 -- broker - list 127.0 . 0.1 : 9092 -- pa
rtition 0
error
this is a test msg from producer
中文
分别查看不同的partition
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - simple -
consumer - shell . bat -- topic mytopic -- offset 0 -- broker - list 127.0 . 0.1 : 9092 -- pa
rtition 0
kafka
error
this is a test msg from producer
中文
终止批处理操作吗( Y / N )? y
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - simple -
consumer - shell . bat -- topic mytopic -- offset 0 -- broker - list 127.0 . 0.1 : 9092 -- pa
rtition 1
go
终止批处理操作吗( Y / N )? y
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows > kafka - simple -
consumer - shell . bat -- topic mytopic -- offset 0 -- broker - list 127.0 . 0.1 : 9092 -- pa
rtition 2
hello
终止批处理操作吗( Y / N )? y
D : \fk\tech\kafka\kafka_2 . 10 - 0.8 . 2.1 \kafka_2 . 10 - 0.8 . 2.1 \b in \windows >
==================================性能测试==================================
1. 测试前准备：创建topic
1.1 查看参数描述：./bin/kafka-topics.sh
1.2 创建topic，根据实际应用场景选择replication-factor和partitions
./bin/kafka-topics.sh --create --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --replication-factor 1 --partitions 1 --topic topic_name
1.3 查看创建topic的详细信息
./bin/kafka-topics.sh --describe --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --topic topic_name
2. producer测试： 主要不同kafka 8部分
2.1 首先看看参数：./bin/kafka-producer-perf-test.sh
usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS --record-size RECORD-SIZE --throughput THROUGHPUT --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...] 脚本规范
optional arguments:
参数producer-props可以并列填写多个参数，这里给个案例：
--producer-props bootstrap.servers=broker_1,broker_2,broker_3 acks=all
这里设定两个参数bootstrap.servers和acks，其他的都是默认，具体代表的意义，可以参看apache kafaka官网的documents>>producer config；
最后列个完整的producer测试脚本
./bin/kafka-producer-perf-test.sh --topic topic_name --throughput 100000 --num-records 100000 --producer-props --producer-props bootstrap.servers=broker_1,broker_2,broker_3 acks=all --record-size 1000
3. consumer测试
需要特别注意一点，就是在测试不同大小的消息队列时，最好使用不同的topic。完整流程最好这样：（1）确定测试的record-size = A1；（2）新建topic；（3）使用producer脚本和（2）的topic，脚本参数设定record-size = A1，产生足量的数据；（4）consumer测试，对应message-size = A1
按照和producer类似的方法，这里只列个脚本案例：
./bin/kafka-consumer-perf-test.sh --messages 100000 --topic topic_name --zookeeper zookeeper_1,zookeeper_2,zookeeper_3 --broker_1,broker_2,broker_3 --message-size A1
本文档介绍了如何在Windows上安装Kafka 0.8.2.1，并提供了创建主题、性能测试（包括producer和consumer）的详细步骤。在测试中，通过kafka-topics.sh创建主题，kafka-producer-perf-test.sh进行producer性能测试，以及kafka-consumer-perf-test.sh进行consumer测试，强调了测试不同消息大小时应使用不同topic。
扫一扫
845
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
