Golang 与Python 连接kafka

最新推荐文章于 2025-02-15 23:37:22 发布

原创最新推荐文章于 2025-02-15 23:37:22 发布 · 428 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#kafka

大数据Spark 专栏收录该内容

16 篇文章

订阅专栏

本文详细介绍了如何在Python和Golang环境中安装并配置Kafka客户端，包括必要的库和依赖项。通过具体示例，展示了如何使用Python和Golang进行消息的生产和消费操作，为开发者提供了实用的Kafka应用指南。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1. 安装

python	golang
pip install kafka-python	git clone https://github.com/edenhill/librdkafka.git cd librdkafka ./configure --prefix=/usr make && make install profile: export PKG_CONFIG_PATH=/usr/lib/pkgconfig go get -u github.com/confluentinc/confluent-kafka-go/kafka

python

golang

pip install kafka-python

git clone https://github.com/edenhill/librdkafka.git

cd librdkafka

./configure --prefix=/usr

make && make install

profile:

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

go get -u github.com/confluentinc/confluent-kafka-go/kafka

编程Producer 连接

Python 编程 Golang编程

from kafka import KafkaProducer     import github.com/confluentinc/confluent-kafka-go/kafka
p = KafkaProducer(                   p,err := kafka.NewProducer(&kafka.ConfigMap{
 bootstrap_servers="master:9092"             "bootstrap.servers": "master"
)                                    })

Producer发送消息:

Python编程 Golang编程(异步发送)

msg = "send msg"                                msg := "send msg"
topic = "job"                                   topic := "job"
p.send(topic, msg)                              diverchan := make(chan kafka.Event)
                                                p.Produce(&kafka.Message{
                                                  TopicPartition: kafka.TopicPartition{
                                                          Topic: &topic,
                                                          Partition:kafka.PartitionAny
                                                  
                                                  },
                                                  Value: []byte(msg)
                                                 }, diverchan)
                                                 e := <- diverchan
                                                 et := e.(*kafka.Message)
                                                 if et.TopicPartition.Error == nil {
                                                            "send successfully"
                                                  }
                                                  close(diverchan)

Consumer编程:

Python编程 Golang编程

from kafka import KafkaConsumer                   c,er:=kafka.NewConsumer(&kafka.ConfigMap{
c = KafkaConsumer("job",                                    "bootstrap.servers": "master",
    bootstrap_servers=["master:9092"]                       "auto.offset.reset": "earliest"
)                                                  })  
for msg in c:                                      c.SubcribeTopics([]string{"job"},nil)
    print msg                                      while true:
                                                        data := c.ReadMessage(-1)
                                                        print data
                                                    c.Close()