Flink cdc connectors RocketMQ

介绍

flink cdc RocketMQ

Flink 版本 1.14.5-SCALA_2.12-JAVA8
发现 apache 官方 有个 RocketMQ-Flink,但是 mvnrepository没有包

编译 cdc jar 包

1. idea 导入 RocketMQ-Flink 项目
2. 修改 pom 依赖 flink 版本相关改成 1.14.5 scala 改成2.12 ,java8
3. 发现 有依赖 schema-registry-client ,新建 idea 项目 导入 rocketmq-schema-registry
4. 编译 install schema-registry-client 到本地 仓库
5. RocketMQ-Flink 依赖 修改完成后不报错的话 package 打包jar
6. copy rocketmq-flink-1.0.0-SNAPSHOT.jar 到 Flink 的工作目录/lib 下
7. 写个flink sql 验证下是否成功

详细步骤

2 步骤内需要修改的内容

       <properties>
        <flink.version>1.14.5</flink.version>
        <scala.binary.version>2.12</scala.binary.version>
       </properties>
       <dependencies>
       <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.rocketmq</groupId>
            <artifactId>schema-registry-client</artifactId>
            <version>${rocketmq.schema.registry.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependencies>

7步骤 验证

CREATE TABLE rocketmq_source (
  `user_action_string` STRING
) WITH (
  'connector' = 'rocketmq',
  'topic' = 'USER_ACTION_LOG_TOPIC',
  'consumerGroup' = 'new_cdc_user-action-group',
  'nameServerAddress' = '192.168.12.198:9876'
);

select * from rocketmq_source;