Kafka安装与集群搭建&基本命令&API

最新推荐文章于 2025-12-29 08:53:01 发布

原创最新推荐文章于 2025-12-29 08:53:01 发布 · 115 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka #hadoop #spark #hdfs

本文详细介绍了Kafka的安装、配置、集群搭建，包括环境变量设置、服务启动与停止、基本命令演示、API使用以及ack机制解析。适合初学者快速上手Kafka。

1、安装与集群搭建

下载并解压安装软件
kafka_2.11-2.0.0.tgz

1.1、配置单机

修改配置文件

vi opt/software/kafka/config/server.properties

broker.id=0
advertised.listeners=PLAINTEXT://192.168.153.141:9092
log.dirs=/opt/bigdata/kafka211/kafka-logs
zookeeper.connect=192.168.153.141:2181
delete.topic.enable=true

添加环境变量并source

export KAFKA_HOME=/opt/software/kafka
export PATH=$PATH:$KAFKA_HOME/bin

source /etc/profile

启动zookeeper

cd /opt/software/zkpr/bin/
./ zkServer.sh start

启动Kafka

//-daemon表示后台启动
kafka-server-start.sh -daemon /opt/software/kafka/config/server.properties

1.2、搭建集群

先搭建zookeeper集群
其他同配置单机，只是server.properties修改不一样

2、基本命令

启动kafka服务

kafka-server-start.sh /opt/bigdata/kafka211/config/server.properties

后台启动

kafka-server-start.sh -daemon /opt/bigdata/kafka211/config/server.properties

关闭Kafka

kafka-server-stop.sh

创建队列

kafka-topics.sh --zookeeper 192.168.153.141:2181 --create --topic mydemo --partitions 3 --replication-factor 1

查看Kafka队列

kafka-topics.sh --zookeeper 192.168.153.141:2181 --list

查看主题详情

kafka-topics.sh --zookeeper 192.168.153.141:2181 --describe --topic mydemo

生产消息到mydemo

kafka-console-producer.sh --topic mydemo --broker-list 192.168.153.141:9092

消费消息(从mydemo中取消息)

kafka-console-consumer.sh --topic mydemo --bootstrap-server 192.168.153.141:9092 --from-beginning

查看队列消息数量

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.153.141:9092 --topic mydemo -time -1 --offsets 1

重置Group topic 初始位置

kafka-streams-application-reset.sh --zookeeper 192.168.153.141:2181 --application-id group_1  --input-topics mydemo

3、API

producer

Properties properties = new Properties();
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.112.100:9092");
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class);
properties.put(ProducerConfig.ACKS_CONFIG,"0");

KafkaProducer<String, String> producer = new KafkaProducer<>(properties);
ProducerRecord<String, String> record = new ProducerRecord<>("mydemo", "message");
producer.send(record);

consumer

Properties prop = new Properties();
prop.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"192.168.112.100:9092");
prop.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
prop.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,StringDeserializer.class);
prop.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,"30000");

//设置是否自动提交，获取数据的状态  false：手动提交；true：自动提交
prop.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,"false");
prop.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG,"1000");

//earliest 从提交的索引处开始消费
//latest 消费最新的消息
prop.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
prop.put(ConsumerConfig.GROUP_ID_CONFIG,"group_1");

KafkaConsumer<String,String> consumer = new KafkaConsumer<>(prop);
consumer.subscribe(Collections.singleton("mydemo2"));
ConsumerRecords<String, String> records = consumer.poll(100);