python2.7简单实现kafka消费和生产信息

最新推荐文章于 2025-03-05 10:03:34 发布

a13838254208

最新推荐文章于 2025-03-05 10:03:34 发布

阅读量599

点赞数

文章标签：大数据 python json

原文链接：http://www.cnblogs.com/LXJ416/p/8177359.html

版权

本文介绍如何使用PyKafka库实现Kafka生产者和消费者功能，详细展示了消息的发送与接收过程，包括配置参数、启动生产者、构造消息、发送消息、创建消费者以及消费消息等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.生产者

from pykafka import KafkaClient
#from kafka import KafkaProducer

import simplejson as json
import logging
import time
import sys




#logging.basicConfig(level = logging.INFO)
client = KafkaClient(hosts = "47.104.108.21:9092" )

topic = client.topics['test']


producer = topic.get_producer()
producer.start()
   
  # 生产消息
msg_dict = {
    "sleep_time": 10,
    "db_config" : {
      "database"  : "test",
      "host"      : "192.168.137.12",
      "user"      : "root",
      "password"  : "root"
    },
    "table"     : "msg",
    "msg"       : "Hello World"
  }
msg = json.dumps(msg_dict)
producer.produce(msg)
producer.stop()

2.消费者

from pykafka import KafkaClient  
host = '47.104.108.21'  
client = KafkaClient(hosts="%s:9092" % host)  
  
print client.topics  
  
  
# 消费者  
topic = client.topics['test']  
consumer = topic.get_simple_consumer(consumer_group='test', auto_commit_enable=True, consumer_id='test')  
for message in consumer:  
    if message is not None:  
        print message.offset, message.value