kafka的安装和配置

最新推荐文章于 2025-06-16 16:51:22 发布
原创 最新推荐文章于 2025-06-16 16:51:22 发布 · 3.9k 阅读 · 1 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#kafka

本文详细介绍了如何在CentOS上安装和配置Kafka，包括从下载Kafka到修改server.properties配置文件，设置broker.id和zookeeper.connect等关键参数，以及启动Kafka服务。接着，文章还涵盖了Kafka的测试步骤，如创建topic、查看topic、发送和接收消息的操作过程。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

一、Kafka的安装和配置

1.kafka的下载地址
http://kafka.apache.org/downloads.html
2.下载后，可以通过winscp，复制到centos环境下，然后解压：
tar zxvf kafka_2.10-0.8.1.1.tgz
3.进入/kafka_2.10-0.8.1.1/config/目录下，对server.properties进行配置：vi server.properties
broker.id=8 这个broker的id,如果是配置的集群的话，这个id需要唯一

port=9092 kafka消息的端口号

host.name kafka主机的ip

zookeeper.connect配置zookeeper的ip，kafka启动的时候会与zookeeper关联，如果zookeeper没有启动就开始启动kafka的话会报错，当然可以配置一个kafka的集群，只关联一个zookeeper，这样也是可行的。
4.启动kafka：进入到kafka的安装目录下，执行命令：
bin/kafka-server-start.sh config/server.properties &

启动成功后出现下面信息：
[1] 18176
[root@logsrv03 kafka_2.10-0.8.1.1]# [2015-08-17 19:52:06,799] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,849] INFO Property broker.id is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,849] INFO Property host.name is overridden to 172.17.6.148 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,849] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,850] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,850] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,850] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,850] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,850] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,851] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,851] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,851] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,851] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,852] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,852] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,852] INFO Property zookeeper.connect is overridden to 172.17.6.142:2181 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,852] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2015-08-17 19:52:06,870] INFO [Kafka Server 8], starting (kafka.server.KafkaServer)
[2015-08-17 19:52:06,872] INFO [Kafka Server 8], Connecting to zookeeper on 172.17.6.142:2181 (kafka.server.KafkaServer)
[2015-08-17 19:52:06,887] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-08-17 19:52:06,894] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:host.name=logsrv03 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.version=1.7.0_71 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.home=/usr/local/jdk1.7.0_71/jre (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.class.path=.:/usr/local/jdk1.7.0_71/lib:/usr/local/jdk1.7.0_71/lib/dt.jar:/usr/local/jdk1.7.0_71/lib/tools.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../clients/build/libs//kafka-clients*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../examples/build/libs//kafka-examples*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/jopt-simple-3.2.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/kafka_2.10-0.8.1.1.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/kafka_2.10-0.8.1.1-javadoc.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/kafka_2.10-0.8.1.1-scaladoc.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/kafka_2.10-0.8.1.1-sources.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/log4j-1.2.15.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/metrics-core-2.2.0.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/scala-library-2.10.1.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/slf4j-api-1.7.2.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/snappy-java-1.0.5.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/zkclient-0.3.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../libs/zookeeper-3.3.4.jar:/usr/local/jiang/kafka_2.10-0.8.1.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,894] INFO Client environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,895] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,895] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,895] INFO Client environment:user.dir=/usr/local/jiang/kafka_2.10-0.8.1.1 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,895] INFO Initiating client connection, connectString=172.17.6.142:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4a1ab1c1 (org.apache.zookeeper.ZooKeeper)
[2015-08-17 19:52:06,916] INFO Opening socket connection to server /172.17.6.142:2181 (org.apache.zookeeper.ClientCnxn)
[2015-08-17 19:52:06,921] INFO Socket connection established to logsrv02/172.17.6.142:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-08-17 19:52:06,941] INFO Session establishment complete on server logsrv02/172.17.6.142:2181, sessionid = 0x14f3b0a17ca0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-08-17 19:52:06,944] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-08-17 19:52:07,221] INFO Found clean shutdown file. Skipping recovery for all logs in data directory '/tmp/kafka-logs' (kafka.log.LogManager)
[2015-08-17 19:52:07,223] INFO Loading log 'test-0' (kafka.log.LogManager)
[2015-08-17 19:52:07,257] INFO Completed load of log test-0 with log end offset 25 (kafka.log.Log)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2015-08-17 19:52:07,280] INFO Loading log 'mytest01-0' (kafka.log.LogManager)
[2015-08-17 19:52:07,282] INFO Completed load of log mytest01-0 with log end offset 1 (kafka.log.Log)
[2015-08-17 19:52:07,283] INFO Loading log 'mytest-0' (kafka.log.LogManager)
[2015-08-17 19:52:07,285] INFO Completed load of log mytest-0 with log end offset 500763 (kafka.log.Log)
[2015-08-17 19:52:07,286] INFO Loading log 'mykafka-1' (kafka.log.LogManager)
[2015-08-17 19:52:07,288] INFO Completed load of log mykafka-1 with log end offset 0 (kafka.log.Log)
[2015-08-17 19:52:07,288] INFO Loading log 'mykafka-0' (kafka.log.LogManager)
[2015-08-17 19:52:07,290] INFO Completed load of log mykafka-0 with log end offset 0 (kafka.log.Log)
[2015-08-17 19:52:07,291] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2015-08-17 19:52:07,294] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2015-08-17 19:52:07,311] INFO Awaiting socket connections on logsrv03:9092. (kafka.network.Acceptor)
[2015-08-17 19:52:07,312] INFO [Socket Server on Broker 8], Started (kafka.network.SocketServer)
[2015-08-17 19:52:07,407] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-08-17 19:52:07,454] INFO conflict in /controller data: {"version":1,"brokerid":8,"timestamp":"1439812327412"} stored data: {"version":1,"brokerid":2,"timestamp":"1439358003358"} (kafka.utils.ZkUtils$)
[2015-08-17 19:52:07,520] INFO Registered broker 8 at path /brokers/ids/8 with address 172.17.6.148:9092. (kafka.utils.ZkUtils$)
[2015-08-17 19:52:07,532] INFO [Kafka Server 8], started (kafka.server.KafkaServer)
[2015-08-17 19:52:07,797] INFO [ReplicaFetcherManager on broker 8] Removed fetcher for partitions [mykafka,0],[mykafka,1],[mytest,0],[mytest01,0] (kafka.server.ReplicaFetcherManager)
[2015-08-17 19:52:07,815] INFO Truncating log mykafka-0 to offset 0. (kafka.log.Log)
[2015-08-17 19:52:07,816] INFO Truncating log mykafka-1 to offset 0. (kafka.log.Log)
[2015-08-17 19:52:07,817] INFO Truncating log mytest-0 to offset 500763. (kafka.log.Log)
[2015-08-17 19:52:07,817] INFO Truncating log mytest01-0 to offset 1. (kafka.log.Log)
[2015-08-17 19:52:07,851] INFO [ReplicaFetcherThread-0-2], Starting  (kafka.server.ReplicaFetcherThread)
[2015-08-17 19:52:07,854] INFO [ReplicaFetcherManager on broker 8] Added fetcher for partitions ArrayBuffer([[mytest,0], initOffset 500763 to broker id:2,host:172.17.6.142,port:9092] , [[mytest01,0], initOffset 1 to broker id:2,host:172.17.6.142,port:9092] , [[mykafka,0], initOffset 0 to broker id:2,host:172.17.6.142,port:9092] ) (kafka.server.ReplicaFetcherManager)
[2015-08-17 19:52:07,873] INFO [ReplicaFetcherManager on broker 8] Removed fetcher for partitions [mykafka,1] (kafka.server.ReplicaFetcherManager)

二、kafka测试

1.创建topic(创建一个名为test的topic)

bin/kafka-topics.sh --create --zookeeper 172.17.6.142:2181 -replication-fator 1 --partitions 1 --topic test

2.查看topic

bin/kafka-topics --list --zookeeper 172.17.6.142:2181

3.发送消息

bin/kafka-console-producer.sh --broker-list 172.17.6.142:9092 --topic test

然后输出要发送的消息
4.接收消息

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
（较详细）Kafka 安装配置
数据爬坡ing的博客
09-13 1698
（较详细）Kafka 安装配置，提供详细步骤，详细代码，注意事项
CDH大数据平台 15Cloudera Manager Console之flume、kafka安装和配置数据采集实战(markdown新版)
80后大叔爱学习
10-09 2021
cloudera manager console实战flume采集kafka数据
kafka详细配置
01-25
kafka详细配置
kafka配置
LiGuanLink的博客
11-19 555
kafka配置生产者配置消费者配置 生产者配置 NAME DESCRIPTION TYPE DEFAULT VALID VALUES IMPORTANCE bootstrap.servers host/port列表，用于初始化建立和Kafka集群的连接。列表格式为host1:port1,host2:port2,…，无需添加所有的集群地址，kafka会根据提供的地址发现其他的地址（你可以多提供几个，以防提供的服务器关闭） list high key.serializer 实现 org.
Spring Boot集成Kafka全攻略：从基础配置到高级实践
最新发布
qq_42773076的博客
06-16 1081
在分布式系统开发中，消息队列是实现系统解耦、异步通信的关键组件，Apache Kafka 凭借其高吞吐量、高可靠性和可扩展性备受青睐。将Kafka集成到Spring Boot项目中，能够快速构建稳定高效的消息处理系统。本文将从依赖添加、配置编写、功能实现等多个维度，深入讲解Spring Boot与Kafka的集成。在文件中添加以下依赖，引入Spring Kafka相关组件以及测试依赖： 二、配置文件 2.1 YAML配置示例 2.2 Properties配置示例 三、核心功能实现 3.1 消息模型 定义
Kafka文档
weixin_30906185的博客
07-06 122
Kafka文档 一、Kafka简介 Kafka是一个分布式的消息队列系统(Message Queue)。 官网：https://kafka.apache.org/ kafka集群有多个Broker服务器组成，每个类型的消息被定义为topic。 同一topic内部的消息按照一定的key和算法被分区(partition)存储在不同的Broker上。 消息生产者producer和消费者...
Kafka 在 Linux 下的集群配置和安装
J老熊
09-26 1866
本文详细介绍了在 Linux 环境中配置和安装 Kafka 集群的过程，包括每个配置项的说明和常见故障的排查方法。通过这些步骤，您可以轻松地构建一个可靠的 Kafka 集群，支持大规模数据流处理和实时数据传输。希望这篇文章能够帮助您顺利搭建和管理 Kafka 集群。
【Kafka】Kafka安装、配置、使用、Java中使用-02
Blue_Pepsi_Cola的博客
04-06 1837
【代码】【Kafka】Kafka安装、配置、使用。
Zookeeper 和kafka的安装和配置1
08-08
安装和配置kafka:1 vim config/ server.properties2 数字改成zookeeper对应的myid数字3 是本机ip,端口不要改4
kafka 配置文件
02-22
server.properties consumer.properties producer.properties
配置Kafka
thewindkee的博客
12-23 550
1.最好是配置成三台，这里我只有2台机器(gk-1,gk-2)，就这样配置了。 2.kafka各个节点配置 ,gk-1节点对应的机器 /etc/hosts如下： 3.  kafka配置文件，server.properties isteners=PLAINTEXT://:9092  zookeeper.connect=localhost:2181 broker.id=0  (其他kaf...
配置kafka
Ezrealmore
05-23 641
broker 参数名称 默认值 说明 auto.create.topics.enable True broker.id 自动生成 delete.topic.enable False 如果为False则管理工具删除主题不会生效 log.retention.bytes -1 删除数据之前的最大大小 log...
kafka的配置详解
航行博客
04-19 715
kafka的配置分为 broker、producter、consumer三个不同的配置 一 BROKER 的全局配置 最为核心的三个配置 broker.id、log.dir、zookeeper.connect 。 ------------------------------------------- 系统 相关 -----------------------------------------...
Kafka 详细配置
Small_Green_Hand的博客
04-29 757
Property Default Description broker.id 每个broker都可以用一个唯一的非负整数id进行标识；这个id可以作为broker的“名字”，并且它的存在使得broker无须混淆consumers就可以迁移到不同的host/port上。你可以选择任意你喜欢的数字作为id，只要id是唯一的即可。 log.dirs /tmp/kafka-logs kafka存放数据的路径。这个路径并不是唯一的，可以是多个，路径之间只需要使用逗号分隔.
kafka配置详解
不甘平庸的人
04-09 5644
kafka的配置分为 broker、producter、consumer三个不同的配置 一 broker 的全局配置 最为核心的三个配置 broker.id、log.dir、zookeeper.connect 。 ------------------------------------------- 系统 相关 ----------------------------------
kafka常用命令及配置参数详解
wyht1222的博客
09-26 2141
kafka常用命令及配置参数详解 常用命令 1、启动zookeeper ​ bin/zookeeper-server-start.sh config/zookeeper.properties & 2、启动kafka bin/kafka-server-start.sh config/server.properties & 3、停止kafka bin/kafka-server-...
kafka集群配置
weixin_44399827的博客
07-05 1256
Kafka作为一个优秀的开源组件，在业界别广泛使用，kafka官方提供了很多配置，供使用者不同的场景下进行配置以提升kafka的效率，合理的配置将直接提升kafka的吞吐量等，而错误的配置则可能引入生产事故等。
Kafka安装和配置
12-28
### 关于 Kafka 的安装和配置 #### 单节点 Kafka 安装过程 对于希望在 Linux 环境下快速启动并运行 Kafka 来说，可以遵循如下说明来完成单节点的安装工作。首先需要准备一台已安装好 Java 运行环境的机器，在此之上下载 Apache Kafka 发布包[^1]。 解压文件后进入 Kafka 所在目录下的 `config` 文件夹内编辑 `server.properties` 文件设置监听器以及 ZooKeeper 地址等参数以便让服务能够正常运作[^5]： ```properties listeners=PLAINTEXT://0.0.0.0:9092 advertised.listeners=PLAINTEXT://<your_external_ip>:9092 log.dirs=/path/to/logs zookeeper.connect=localhost:2181 ``` 上述配置项分别指定了内部与外部访问使用的协议及端口号、日志存储路径还有用于协调各组件间通信的ZooKeeper实例位置。 接着通过命令行工具执行脚本来启动 Kafka Broker 和内置的 ZooKeeper 实例（如果选择不使用独立部署版的话），具体操作如下所示[^4]: ```bash ./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties ./bin/kafka-server-start.sh -daemon config/server.properties ``` 此时便完成了基本的服务架设流程；为了验证一切顺利无误，则可利用官方提供的辅助程序创建测试 Topic 并发送消息给它尝试接收查看效果如何[^3]。 #### 使用 Admin API 对 Kafka 进行管理 除了手动编写 Shell 脚本外还可以借助 Admin Client 提供的功能接口实现更加灵活高效的运维作业模式——比如增删改查 Topics 或者调整 Brokers 参数等等动作均能被封装成函数调用来简化日常管理工作量[^2]。 例如要获取某个特定主题的信息就可以这样写 Python 代码片段: ```python from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient(bootstrap_servers="localhost:9092", client_id='test') topics = admin_client.describe_topics(["my_topic"]) print(topics) ``` 这段示范展示了怎样建立到集群的连接并且请求有关指定名称的主题详情记录返回给用户查阅。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值