flink-cdc(java案例)

最新推荐文章于 2025-09-09 03:18:56 发布

原创

最新推荐文章于 2025-09-09 03:18:56 发布 · 7.6k 阅读

14 ·

CC 4.0 BY-SA版权

文章标签：

#java #flink #maven

本文介绍如何使用 Flink CDC 进行 MySQL 数据库的实时变更数据捕获，并将其同步到 Kafka 中。涵盖 Maven 依赖配置、MySQL binlog 启用、自定义序列化器实现、Kafka 连接及 Flink 作业设置等关键步骤。

flinkcdc是一款flink的开源项目，他继承了传统的cdc工具，让实时开发更适合flink语言详情见下：

1：开发前的准备

cdc开发所需要的maven依赖-切记flink版本的更新导致不同的工具在使用时候会存在不兼容的问题

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.lkr.flink</groupId>
    <artifactId>flink-cdc</artifactId>
    <version>1.0</version>
    <packaging>jar</packaging>

    <name>Flink Quickstart Job</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <flink.version>1.13.5</flink.version>
        <target.java.version>1.8</target.java.version>
        <scala.binary.version>2.12</scala.binary.version>
        <log4j.version>2.12.1</log4j.version>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <mysql.version>5.1.49</mysql.version>
        <flinkcdc.version>2.0.0</flinkcdc.version>
        <fastjson.version>1.2.75</fastjson.version>
    </properties>

    <repositories>
        <repository>
            <id>apache.snapshots</id>
            <name>Apache Development Snapshot Repository</name>
            <url>https://repository.apache.org/content/repositories/snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_${scala.binary.version}</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.8.0</version>
        </depen

最低0.47元/天解锁文章