kafka配置

最新推荐文章于 2024-07-26 16:48:44 发布

AlanNote

最新推荐文章于 2024-07-26 16:48:44 发布

阅读量175

点赞数

CC 4.0 BY-SA版权

分类专栏：消息队列文章标签： kafka

本文链接：https://blog.youkuaiyun.com/u012588542/article/details/83823378

消息队列专栏收录该内容

1 篇文章

订阅专栏

本文详细解析了Kafka配置文件中的关键参数，如listeners和advertised.listeners的作用及相互关系，指导如何配置Kafka监听所有网络接口。同时，介绍了如何在同一台机器上搭建伪分布式Kafka集群，包括创建配置文件、修改必要参数以及启动多broker实例的过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

kafka_1.1.1 配置注意事项

kafka broker默认配置文件似乎使kafka只监听loopback接口的端口，为了使用kafka监听所有网络接口的对应端口，需要修改默认配置文件，配置文件中关于socket相关的选项有两个，listeners和advertised.listeners，每个选项作用及相互关系需理解。

  ############################# Socket Server Settings #############################
 
 25 # The address the socket server listens on. It will get the value returned from
 26 # java.net.InetAddress.getCanonicalHostName() if not configured.
 27 #   FORMAT:
 28 #     listeners = listener_name://host_name:port
 29 #   EXAMPLE:
 30 #     listeners = PLAINTEXT://your.host.name:9092
 31 #listeners=PLAINTEXT://:9092
 32
 33 # Hostname and port the broker will advertise to producers and consumers. If not set,
 34 # it uses the value for "listeners" if configured.  Otherwise, it will use the value
 35 # returned from java.net.InetAddress.getCanonicalHostName().
 36 #advertised.listeners=PLAINTEXT://your.host.name:9092
 37
 38 # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
 39 #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
 40
 41 # The number of threads that the server uses for receiving requests from the network and sending responses to the network
 42 num.network.threads=3
 43
 44 # The number of threads that the server uses for processing requests, which may include disk I/O
 45 num.io.threads=8
 46
 47 # The send buffer (SO_SNDBUF) used by the socket server
 48 socket.send.buffer.bytes=102400
 49
 50 # The receive buffer (SO_RCVBUF) used by the socket server
 51 socket.receive.buffer.bytes=102400
 52
 53 # The maximum size of a request that the socket server will accept (protection against OOM)
 54 socket.request.max.bytes=104857600

在同一台机器上启动多个broker实例（伪分布式），应该为每个broker实例分配一个配置文件，可使用默认文件副本，根据kafka quick start教程，当中某些项必须修改。包括broker.id、listeners、log.dirs等。

kafka自带java客户端常用命令

1.启动服务器
#启动zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
#启动Kafka
bin/kafka-server-start.sh config/server.properties

2.创建主题
#创建
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
#查看所有主题
bin/kafka-topics.sh --list --zookeeper localhost:2181

3.发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

4.启动消费者
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

5.设置多broker的Kafka集群
#为每个broker创建配置文件
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
#编辑这些配置文件
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1

config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2
#启动这两个节点
bin/kafka-server-start.sh config/server-1.properties &
bin/kafka-server-start.sh config/server-2.properties &
#创建一个新主题，副本为3
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

#停止进程节点
ps aux | grep server-1.properties
kill -9 xxxx

gwet curl ??

ps aux | grep server-1.properties