Spring集成Kafka 结合Kafka Tool使用

最新推荐文章于 2025-03-22 01:47:58 发布

Amy安

最新推荐文章于 2025-03-22 01:47:58 发布

阅读量1.9k

点赞数

分类专栏： Java 文章标签：中缀表达式 kafka spring

本文链接：https://blog.youkuaiyun.com/u014294325/article/details/108324941

版权

Java 专栏收录该内容

6 篇文章

订阅专栏

下载安装

1.从http://kafka.apache.org/downloads 下载kafka
2.从https://zookeeper.apache.org/releases.html 下载zookeeper
3.从http://www.kafkatool.com/download.html 下载kafkatool

注意事项

kafka有路径长度限制，放置位置不宜过长

配置

Spring项目

pox.xml

		<dependency>
			<groupId>org.springframework.kafka</groupId>
			<artifactId>spring-kafka</artifactId>
		</dependency>

kafka

kafka/config/server.properties
修改listeners listeners=PLAINTEXT://127.0.0.1:9092
修改log.dirslog.dirs=D:\\kafka-logs

kafka/config/zookeeper.properties
修改dataDirdataDir=D:\\zookeeper-logs

zookeeper

zookeeper/conf/zoo_sample.cfg
修改dataDirdataDir=D:\\tmp\\zkzookeeper

启动

1.启动kafka/bin内的zookeeper-server-start
zookeeper-server-start.bat ../../config/zookeeper.properties

2.启动zookeeper/bin内的zkServer
zkServer

3.启动kafka
kafka-server-start.bat ../../config/server.properties

4.创建topic
例名为testk
kafka-topics.bat --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic testk

注意事项

Kafka启动报错

Unable to parse hostname:9092 to a broker endpoint

[2020-08-31 16:24:39,080] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-08-31 16:24:39,887] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-08-31 16:24:39,917] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from '127.0.0.1:9092': Unable to parse 127.0.0.1:9092 to a broker endpoint
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:268)
        at kafka.server.KafkaConfig.$anonfun$listeners$1(KafkaConfig.scala:1633)
        at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1632)
        at kafka.server.KafkaConfig.advertisedListeners(KafkaConfig.scala:1660)
        at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1731)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1709)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1273)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1221)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:29)
        at kafka.Kafka$.main(Kafka.scala:68)
        at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse 127.0.0.1:9092 to a broker endpoint
        at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:57)
        at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:265)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:273)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:265)
        ... 11 more

原因：server.properties中listeners或advertised.listeners的格式有误。

尝试解决方案
主机名不能有非法字符 _(下横杠) .（点），有的话要修改后重启。
在这里插入图片描述

检查格式
[协议名称]+冒号+双斜杠+[地址]+冒号+[端口号]

#   FORMAT:
     listeners = listener_name://host_name:port
#   EXAMPLE:
     listeners = PLAINTEXT://your.host.name:9092

No security protocol defined

多为协议名称错误
参照
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

[2020-08-31 17:06:20,286] INFO Client environment:java.io.tmpdir=C:\Users\DELL\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,287] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,287] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,287] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,288] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,288] INFO Client environment:user.name=DELL (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,288] INFO Client environment:user.home=C:\Users\DELL (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,289] INFO Client environment:user.dir=D:\Kafka\bin\windows (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,289] INFO Client environment:os.memory.free=979MB (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,290] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,290] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,326] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@13bc8645 (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:20,347] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-08-31 17:06:20,375] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-08-31 17:06:20,378] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-08-31 17:06:20,390] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-08-31 17:06:20,392] INFO Socket connection established, initiating session, client: /127.0.0.1:55399, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2020-08-31 17:06:38,381] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-08-31 17:06:38,393] WARN Client session timed out, have not heard from server in 18000ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-08-31 17:06:38,515] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-08-31 17:06:38,516] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-08-31 17:06:38,518] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-08-31 17:06:38,521] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2020-08-31 17:06:38,525] INFO shutting down (kafka.server.KafkaServer)
[2020-08-31 17:06:38,549] INFO shut down completed (kafka.server.KafkaServer)
[2020-08-31 17:06:38,550] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-08-31 17:06:38,552] INFO shutting down (kafka.server.KafkaServer)

可能原因：
1.无目录访问权限多为log.dirs 调整即可
2.zookeeper地址有误调整即可

使用

@RestController
public class DemoController {

    @Autowired
    private KafkaTemplate<String, String> kafka;

    @RequestMapping("/send/{topic}/{key}/{data}")
    @ResponseBody
    String send(@PathVariable String topic, @PathVariable String key, @PathVariable String data) {
        kafka.send(topic, key, data);
        return "success";
    }

    @KafkaListener(id = "testk", topics = "testk")
    public void listener(ConsumerRecord<?, ?> cr) {
        System.out.println("listen==>"+String.format("%s - %s : %s", cr.topic(), cr.key(), cr.value()));
    }

}

http://localhost:22746/send/testk/a/a1
http://localhost:22746/send/testk/b/b1

listen==>testk - a : a1
listen==>testk - b : b1

Kafka Tool

启动并设置连接
在这里插入图片描述

查看消息