kafka基本命令，python-kafka使用

最新推荐文章于 2025-04-15 10:14:22 发布

weixin_34356310

最新推荐文章于 2025-04-15 10:14:22 发布

阅读量244

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 python json

原文链接：http://blog.51cto.com/haoyonghui/2383373

本文详细介绍了Kafka的启动流程，包括使用nohup命令后台启动Kafka服务，以及如何通过kafka-console-consumer工具查看消费者消息。此外，还深入探讨了Kafka主题的描述方法，并提供了生产者和消费者类的实现代码，帮助读者理解如何在Python中发送和接收Kafka消息。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.启动
nohup kafka-server-start /usr/local/etc/kafka/server-1.properties &

2.查看消费者
kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

3.描述
kafka-topics --describe --bootstrap-server localhost:9092 --topic my-replicated-topic

生产者:
class Kafkaproducer():
"""
生产模块：根据不同的key,区分消息
"""

def __init__(self, kafka_host,kafka_port,kafka_topic, key):
    self.kafka_host = kafka_host
    self.kafka_port = kafka_port
    self.kafka_topic = kafka_topic
    self.key = key
    bootstrap_servers = "{kafka_host}:{kafka_port}".format(
        kafka_host=self.kafka_host,kafka_port=self.kafka_port
    )

    self.producer = KafkaProducer(bootstrap_servers = [bootstrap_servers])

def send_message(self,params):
    try:
        params_message = json.dumps(params).encode('utf-8')
        producer = self.producer
        #producer.send(self.kafka_topic, key=self.key, value=params_message)
        producer.send(self.kafka_topic,key=self.key,value=params_message) #key,value必须是字节
        producer.flush()

    except KafkaError as e:
        print(e)

消费者:
class Kafkaconsumer():
"""
消费者
"""
def init(self,kafka_host,kafka_port, kafka_topic, groupid):
self.kafka_host = kafka_host
self.kafka_port = kafka_port
self.kafka_topic = kafka_topic
self.groupid = groupid
self.consumer = KafkaConsumer(self.kafka_topic,group_id=self.groupid,
bootstrap_servers = "{kafka_host}:{kafka_port}".format(
kafka_host=self.kafka_host,
kafka_port=self.kafka_port
))

def consume_message(self):
    try:
        for message in self.consumer:
            yield message
    except KeyboardInterrupt as e:
        print(e)

转载于:https://blog.51cto.com/haoyonghui/2383373