kafka实践

最新推荐文章于 2024-09-04 14:14:04 发布

abcd1101

最新推荐文章于 2024-09-04 14:14:04 发布

阅读量939

点赞数 1

CC 4.0 BY-SA版权

分类专栏： kafka 文章标签： kafka

本文链接：https://blog.youkuaiyun.com/abcd1101/article/details/85872057

kafka 专栏收录该内容

2 篇文章

订阅专栏

这篇博客详细介绍了Kafka的部署过程，包括启动Zookeeper和Kafka服务器，创建主题，列出主题，发送和消费消息。此外，还提到了可能出现的问题及解决办法，如JVM内存设置和清理offset的注意事项。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

https://gitee.com/abcd_1101/BigData/tree/master/springboot-kafka-demo

这里的readme有详细步骤，下包什么的就不说了，全是命令加代码。

1.startup zookeeper
nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
netstat -tunlp |grep 2181
2.startup kafka
nohup bin/kafka-server-start.sh config/server.properties &
netstat -tunlp |grep 9092

1.create topic:
[root@VM_0_14_centos kafka_2.11-2.1.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
[2018-12-26 12:27:52,627] INFO Accepted socket connection from /127.0.0.1:57358 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-12-26 12:27:52,633] INFO Client attempting to establish new session at /127.0.0.1:57358 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-12-26 12:27:52,638] INFO Established session 0x10303ac9e570001 with negotiated timeout 30000 for client /127.0.0.1:57358 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-12-26 12:27:53,062] INFO Got user-level KeeperException when processing sessionid:0x10303ac9e570001 type:setData cxid:0x4 zxid:0x1f txntype:-1 reqpath:n/a Error Path:/config/topics/test Error:KeeperErrorCode = NoNode for /config/topics/test (org.apache.zookeeper.server.PrepRequestProcessor)
Created topic "test".
[2018-12-26 12:27:53,186] INFO Processed session termination for sessionid: 0x10303ac9e570001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-12-26 12:27:53,227] INFO Closed socket connection for client /127.0.0.1:57358 which had sessionid 0x10303ac9e570001 (org.apache.zookeeper.server.NIOServerCnxn)
[2018-12-26 12:27:53,319] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(test-0) (kafka.server.ReplicaFetcherManager)
[2018-12-26 12:27:53,422] INFO [Log partition=test-0, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log)
[2018-12-26 12:27:53,429] INFO [Log partition=test-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 73 ms (kafka.log.Log)
[2018-12-26 12:27:53,431] INFO Created log for partition test-0 in /tmp/kafka-logs with properties {compression.type -> producer, message.format.version -> 2.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2018-12-26 12:27:53,432] INFO [Partition test-0 broker=0] No checkpointed highwatermark is found for partition test-0 (kafka.cluster.Partition)
[2018-12-26 12:27:53,447] INFO Replica loaded for partition test-0 with initial high watermark0 (kafka.cluster.Replica)
[2018-12-26 12:27:53,450] INFO [Partition test-0 broker=0] test-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)

2.list topic
bin/kafka-topics.sh --list --zookeeper 119.29.56.220:2181
[root@VM_0_14_centos kafka_2.11-2.1.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
[2018-12-26 12:28:13,172] INFO Accepted socket connection from /127.0.0.1:57376 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-12-26 12:28:13,172] INFO Client attempting to establish new session at /127.0.0.1:57376 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-12-26 12:28:13,180] INFO Established session 0x10303ac9e570002 with negotiated timeout 30000 for client /127.0.0.1:57376 (org.apache.zookeeper.server.ZooKeeperServer)
test
[2018-12-26 12:28:13,233] INFO Processed session termination for sessionid: 0x10303ac9e570002 (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-12-26 12:28:13,249] INFO Closed socket connection for client /127.0.0.1:57376 which had sessionid 0x10303ac9e570002 (org.apache.zookeeper.server.NIOServerCnxn)
上面原来是提示zookeeper出错了，所以需要重新启动zookeeper，正常只会出现test

3.produce message
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>This is a message
>This is another message

4.consume topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

notice:
1.kafka default jvm is 1G, you can change bin/kafka-server-start.sh to change JVM
2.if want to use spring connect, update config/server.properties
listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners：PLAINTEXT://119.29.56.220:9092
3.Error while reading checkpoint file /tmp/kafka-logs/cleaner-offset-checkpoint
NoSuchFileException: /tmp/kafka-logs/cleaner-offset-checkpoint
每次清理完,要更新当前已经清理到的位置, 记录在cleaner-offset-checkpoint文件中,作为下一次清理时生成firstDirtyOffset的参考;