kafka的docker安装和生产者、消费者的demo

备注：请将your_server_ip替换为腾讯云服务器的公网IP地址

1、在服务上安装容器

首先，创建一个新的文件夹用于存放docker-compose配置文件，然后在该文件夹下创建一个docker-compose.yml文件，内容如下：

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://your_server_ip:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

在该目录下运行docker-compose up -d启动Kafka和Zookeeper容器。

2、PyCharm中编写Kafka程序

2.1、安装Python Kafka库：在你的项目环境中安装kafka-python库，可以通过pip安装：

pip install kafka-python

2.2、编写Producer代码：下面是一个简单的Kafka生产者示例，用于发送消息到Kafka主题。

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='your_server_ip:9092')
producer.send('test-topic', b'Hello, Kafka!')
producer.flush()
producer.close()

2.3、编写Consumer代码：同样，这里有一个简单的消费者示例，用于接收来自Kafka主题的消息。

from kafka import KafkaConsumer
consumer = KafkaConsumer('test-topic', bootstrap_servers='your_server_ip:9092', auto_offset_reset='earliest')
for msg in consumer:
    print(msg.value)

最后可以看到消费者的消息

----------------------------------------

例子2

producer

import json

from kafka import KafkaProducer
import uuid

producer = KafkaProducer(bootstrap_servers='your_server_ip:9092')

order = {
    "order_id": str(uuid.uuid4()),
    "user": "strive",
    "item": "mushroom pizza",
    "quantity": 2
}

value = json.dumps(order).encode('utf-8')

producer.send('orders', value)
producer.flush()
producer.close()

consumer

import json

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers='your_server_ip:9092',
    group_id='order-tracker',
    auto_offset_reset='earliest',
)

consumer.subscribe(['orders'])
print("Consumer is running and subscribed to orders topic")

try:
    while True:
        messages = consumer.poll(1.0)  # 这里返回的是 dict 类型
        if not messages:
            continue

        for topic_partition, msgs in messages.items():
            for msg in msgs:
                try:
                    value = msg.value.decode('utf-8')
                    print(value)
                    order = json.loads(value)
                    print(f"Order received: {order['quantity'] * order['item']} from {order['user']}")
                except Exception as e:
                    print(f"Error processing message: {e}")
                    continue

except KeyboardInterrupt:
    print("Stopping consumer")
except Exception as e:
    print(f"Consumer error: {e}")
finally:
    consumer.close()

运行结果