<think>我们注意到用户意图是使用librdkafka库连接华为云MRS Kafka服务(带Kerberos认证)。下面将提供详细步骤:
### 一、环境准备
1. **安装librdkafka**
从源码编译安装(确保系统已安装openssl、cyrus-sasl等依赖):
```bash
# 安装依赖
sudo yum install -y gcc-c++ make openssl-devel cyrus-sasl-gssapi krb5-devel
# 下载并编译
wget https://github.com/edenhill/librdkafka/archive/refs/tags/v1.9.2.tar.gz
tar -xvf v1.9.2.tar.gz
cd librdkafka-1.9.2
./configure --enable-sasl --enable-ssl
make
sudo make install
```
2. **获取MRS集群认证文件**
从华为云MRS集群管理页面下载:
- `krb5.conf`:Kerberos客户端配置
- `user.keytab`:用户密钥文件
- `client.jks`:Kafka SSL信任证书
### 二、配置Kerberos认证
1. **配置krb5.conf**
将下载的`krb5.conf`复制到`/etc/krb5.conf`(需root权限)或设置环境变量:
```bash
export KRB5_CONFIG=/path/to/your/krb5.conf
```
2. **初始化Kerberos票据**
使用keytab获取票据(注意替换主体名称):
```bash
kinit -kt /path/to/user.keytab username@HADOOP.COM
klist # 验证票据是否有效
```
### 三、编写生产者示例(C语言)
```c
#include <librdkafka/rdkafka.h>
int main() {
rd_kafka_conf_t *conf = rd_kafka_conf_new();
// 设置基础参数
rd_kafka_conf_set(conf, "bootstrap.servers", "192.168.0.1:21007", NULL, 0);
rd_kafka_conf_set(conf, "security.protocol", "SASL_SSL", NULL, 0);
rd_kafka_conf_set(conf, "sasl.kerberos.service.name", "kafka", NULL, 0);
rd_kafka_conf_set(conf, "ssl.ca.location", "/path/to/client.jks", NULL, 0); // 注意:librdkafka要求PEM格式,需转换JKS
// 创建生产者
rd_kafka_t *producer = rd_kafka_new(RD_KAFKA_PRODUCER, conf, NULL, 0);
// 发送消息
rd_kafka_producev(producer,
RD_KAFKA_V_TOPIC("test_topic"),
RD_KAFKA_V_VALUE("Hello, MRS Kafka!", 18),
RD_KAFKA_V_END);
rd_kafka_flush(producer, 10*1000); // 等待发送完成
rd_kafka_destroy(producer);
return 0;
}
```
> **注意**:librdkafka要求SSL证书为PEM格式,需转换JKS:
> ```bash
> keytool -importkeystore -srckeystore client.jks -destkeystore client.p12 -deststoretype PKCS12
> openssl pkcs12 -in client.p12 -out client.pem -nodes
> ```
### 四、编译与运行
1. **编译程序**
```bash
gcc -o producer producer.c -lrdkafka -lssl -lcrypto -lsasl2
```
2. **运行前配置SASL**
设置环境变量启用GSSAPI认证:
```bash
export KRB5_CONFIG=/path/to/krb5.conf
export KAFKA_OPTS="-Djava.security.krb5.conf=/path/to/krb5.conf"
```
### 五、消费者示例(关键参数)
```c
rd_kafka_conf_set(conf, "group.id", "test_group", NULL, 0);
rd_kafka_conf_set(conf, "auto.offset.reset", "earliest", NULL, 0);
// 创建消费者
rd_kafka_t *consumer = rd_kafka_new(RD_KAFKA_CONSUMER, conf, NULL, 0);
rd_kafka_subscribe(consumer, topics); // topics为topic列表
// 消费消息
rd_kafka_message_t *msg;
while (running) {
msg = rd_kafka_consumer_poll(consumer, 1000);
if (msg) {
printf("Received: %.*s\n", (int)msg->len, (char *)msg->payload);
rd_kafka_message_destroy(msg);
}
}
```
### 六、常见问题解决
1. **SASL认证失败**
- 检查`klist`确保票据有效
- 设置调试日志:`rd_kafka_conf_set(conf, "debug", "security", NULL, 0);`
2. **SSL证书问题**
- 确认`ssl.ca.location`指向正确的PEM文件
- 关闭证书验证(仅测试环境):
```c
rd_kafka_conf_set(conf, "enable.ssl.certificate.verification", "false", NULL, 0);
```
3. **连接超时**
- 检查MRS Kafka安全组是否开放21007端口
- 确认VPC网络互通
> 完整示例参考华为云官方文档《Kafka客户端使用说明》[^2]
§§ 相关问题 §§
1. 如何将JKS格式证书转换为PEM格式?
2. librdkafka如何开启Kerberos调试日志?
3. 华为云MRS Kafka的SASL_SSL端口是多少?
4. 如何在C++项目中集成librdkafka?
5. 消费者如何提交手动偏移量?</think>### 使用 librdkafka 连接华为云 MRS Kafka 的开发指南
华为云 MRS Kafka 采用 Kerberos 认证的安全模式(基于 Hadoop 3.3.1/Kafka 2.4),以下是使用 C/C++ 的 librdkafka 库进行开发的完整流程:
---
#### 一、环境准备
1. **安装依赖库**
```bash
# CentOS 示例
sudo yum install -y gcc-c++ make openssl-devel cyrus-sasl-gssapi krb5-devel
```
2. **安装 librdkafka**
```bash
wget https://github.com/edenhill/librdkafka/archive/refs/tags/v2.2.0.tar.gz
tar -xzf v2.2.0.tar.gz
cd librdkafka-2.2.0
./configure --enable-sasl --enable-ssl
make
sudo make install
```
3. **获取 MRS 认证文件**(从集群管理页面下载)
- `krb5.conf`:Kerberos 客户端配置
- `user.keytab`:用户密钥文件
- `client.jks`:Kafka SSL 信任证书(需转换为 PEM 格式)
```bash
keytool -importkeystore -srckeystore client.jks -destkeystore client.p12 -deststoretype PKCS12
openssl pkcs12 -in client.p12 -out client.pem -nodes
```
---
#### 二、生产者示例代码(C++)
```cpp
#include <librdkafka/rdkafkacpp.h>
int main() {
// 1. 创建配置
RdKafka::Conf *conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);
// 2. 设置关键参数
conf->set("bootstrap.servers", "192.168.0.1:21007", errstr); // 替换为 MRS Kafka 地址
conf->set("security.protocol", "sasl_ssl", errstr);
conf->set("sasl.kerberos.service.name", "kafka", errstr);
conf->set("ssl.ca.location", "/path/to/client.pem", errstr); // 转换后的证书
// 3. 配置 Kerberos
conf->set("sasl.kerberos.keytab", "/path/to/user.keytab", errstr);
conf->set("sasl.kerberos.principal", "username@HADOOP.COM", errstr);
// 4. 创建生产者实例
RdKafka::Producer *producer = RdKafka::Producer::create(conf, errstr);
// 5. 发送消息
std::string topic = "test_topic";
std::string payload = "Hello MRS Kafka!";
producer->produce(topic, RdKafka::Topic::PARTITION_UA,
RdKafka::Producer::RK_MSG_COPY,
const_cast<char*>(payload.c_str()), payload.size(),
NULL, NULL);
producer->flush(10*1000); // 等待发送完成
delete producer;
}
```
---
#### 三、消费者示例代码(C)
```c
#include <librdkafka/rdkafka.h>
int main() {
rd_kafka_conf_t *conf = rd_kafka_conf_new();
// 设置连接参数
rd_kafka_conf_set(conf, "bootstrap.servers", "192.168.0.1:21007", NULL, 0);
rd_kafka_conf_set(conf, "security.protocol", "sasl_ssl", NULL, 0);
rd_kafka_conf_set(conf, "sasl.kerberos.service.name", "kafka", NULL, 0);
// 配置 Kerberos
rd_kafka_conf_set(conf, "sasl.kerberos.keytab", "/path/to/user.keytab", NULL, 0);
rd_kafka_conf_set(conf, "sasl.kerberos.principal", "username@HADOOP.COM", NULL, 0);
// 创建消费者
rd_kafka_t *consumer = rd_kafka_new(RD_KAFKA_CONSUMER, conf, NULL, 0);
rd_kafka_subscribe(consumer, (const char **){"test_topic"}, 1);
// 消费消息
while (1) {
rd_kafka_message_t *msg = rd_kafka_consumer_poll(consumer, 1000);
if (msg) {
printf("Received: %.*s\n", (int)msg->len, (char *)msg->payload);
rd_kafka_message_destroy(msg);
}
}
}
```
---
#### 四、编译与运行
```bash
# 编译生产者 (C++)
g++ -o producer producer.cpp -lrdkafka++ -lrdkafka -lssl -lcrypto -lsasl2
# 编译消费者 (C)
gcc -o consumer consumer.c -lrdkafka -lssl -lcrypto -lsasl2
# 运行前配置 Kerberos
export KRB5_CONFIG=/path/to/krb5.conf
kinit -kt /path/to/user.keytab username@HADOOP.COM
# 执行程序
./producer
./consumer
```
---
#### 五、关键配置说明
| **配置项** | **值** | **作用** |
|------------|--------|----------|
| `bootstrap.servers` | `IP:21007` | MRS Kafka 地址(端口固定为 21007)[^1] |
| `security.protocol` | `sasl_ssl` | 启用 SASL 和 SSL 加密 |
| `sasl.kerberos.keytab` | 密钥文件路径 | Kerberos 认证密钥 |
| `ssl.ca.location` | PEM 文件路径 | 集群 SSL 证书 |
> **调试提示**:设置环境变量 `export RD_KAFKA_LOG_LEVEL=7` 可输出详细调试日志。
---
#### 六、常见问题解决
1. **认证失败 (SASL 错误)**
- 检查 `klist` 确保票据有效
- 确认 `krb5.conf` 中的域名与 MRS 集群一致
2. **SSL 握手失败**
- 确保证书已正确转换为 PEM 格式
- 添加配置:`conf->set("enable.ssl.certificate.verification", "false", errstr);`(仅测试环境)
3. **连接超时**
- 检查安全组是否开放 21007 端口
- 确认 VPC 网络互通
> 完整配置参考:[华为云 Kafka 全景实践文档](https://support.huaweicloud.com/bestpractice-kafka/kafka-bp-0002.html)[^2]