Kafka消息队列基础入门和实战例子

1、Kafka

1.1 Kafka部署配置

1.1.1 下载Kafka

下载Kafka https://kafka.apache.org/downloads.html

https://archive.apache.org/dist/kafka/2.4.1/kafka_2.11-2.4.1.tgz

下载Scala-2.11版本

Scala-2.11经典版本

解压

直接解压到某个目录，可以一起放在Java相关的目录下，不需要额外安装

3.1.2 修改配置文件

进入Config目录，修改配置文件

修改zookeeper.properties

==> zookeeper启动后会自动创建zk-data文件夹，保存所需数据。

clientPort=2181

zookeeper的端口号配置

修改server.properties文件

advertised.listeners=PLAINTEXT://localhost:9092

zookeeper.connect=localhost:2181

运行Kafka

进入/bin/windows目录

启动zookeeper

zookeeper-server-start.bat ../../config/zookeeper.properties

启动Kafka

kafka-server-start.bat ../../config/server.properties

订阅Kafka消息

kafka-console-consumer --bootstrap-server localhost:9092 --topic TEST-TOPIC

Python写个Kafka生产者测试一下

pip install kafka-python

代码

# -*- coding:utf-8 -*-
import json
from kafka import KafkaProducer

if __name__ == '__main__':
    print("Kafka生产消息")
    producer = KafkaProducer(
        value_serializer=lambda v: json.dumps(v).encode('utf-8'),
        bootstrap_servers=['localhost:9092']
    )

    msgDict = {
        "id": 1,
        "name": "Kafka Producer",
        "msg": "Test Kafka Producer",
        "xList": [1, 2, 3]
    }

    producer.send("TEST-TOPIC", msgDict)
    producer.close()

Console运行截图

Kafka生产消息

Process finished with exit code 0