Apache Kafka 的安装与使用

最新推荐文章于 2025-10-17 16:12:57 发布

原创最新推荐文章于 2025-10-17 16:12:57 发布 · 1k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#kafka-消息发布与订阅 #big-data

BIGDATA 专栏收录该内容

35 篇文章

订阅专栏

本文详细介绍如何在Linux环境下安装并配置Kafka集群，包括单节点配置与多节点集群搭建过程，并提供了创建主题、启动生产者与消费者以及进行故障容错测试的方法。

安装Java

> sudo apt-get install openjdk-8-jdk

安装Kafka

# Download
> wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> tar -xzf kafka_2.11-0.10.2.0.tgz
> cd kafka_2.11-0.10.2.0

# start a ZooKeeper server
> ./bin/zookeeper-server-start.sh config/zookeeper.properties

# start the Kafka server:
> ./bin/kafka-server-start.sh config/server.properties

# Create a topic
> ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

# list topic
> ./bin/kafka-topics.sh --list --zookeeper localhost:2181

# Delete a Topic
> vim server.properties
    delete.topic.enable=true
> ./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test1

# Run the producer:
> ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

# Start a consumer
> ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Setting up a multi-broker cluster

# copy config
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.properties

# edit config
> vim config/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1

> vim config/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2

note: The broker.id property is the unique and permanent name of each node in the cluster.

# Running node
> ./bin/kafka-server-start.sh config/server-1.properties &
> ./bin/kafka-server-start.sh config/server-2.properties &

# Create a new topic with a replication factor
> ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic my-replicated-topic

# Describe topics
> ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic

# Start procedure:
> ./bin/kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --sync --topic my-replicated-topic

# Start Consumer:
> ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094 --from-beginning --topic my-replicated-topic

Fault-tolerance Test

# we kill cluster leader 1, system should auto randomly to a new cluster node.
> sudo netstat -ntlp
> kill -9 10598