Debezium实战-第4章 连接器-Mysql

本文详细介绍了Debezium的MySQL连接器如何在数据库架构中工作,包括支持的MySQL部署类型、快照机制、数据变更事件的处理方式,以及连接器的配置和设置。连接器通过读取MySQL的二进制日志来跟踪数据变更,生成Kafka主题事件。文章还讨论了连接器如何处理初始快照和增量快照,以及如何在不同场景下配置和处理数据类型映射。此外,还涵盖了连接器在故障情况下的行为和监控指标。

4.2 Mysql

MySQL 有一个二进制日志(binlog),它按照提交到数据库的顺序记录所有操作。这包括对表Schema的更改以及对表中数据的更改。MySQL 使用 binlog 进行复制和恢复。

该Debezium的MySQL连接器读取二进制日志,产生变化的事件行级INSERTUPDATEDELETE操作,并发出变更事件Kafka主题topics。客户端应用程序读取这些 Kafka 主题。

由于 MySQL 通常设置为在指定的时间段后清除 binlog,因此 MySQL 连接器会为您的每个数据库执行产生一个初始一致的快照。MySQL 连接器从创建快照的点读取二进制日志。

连接器的工作原理

连接器支持MySQL的布署,对于规划您的应用程序很有用。要优化配置和运行 Debezium MySQL 连接器,了解连接器如何跟踪表结构、公开schema变更、执行快照和确定 Kafka 主题名称会很有帮助。

注意
Debezium MySQL 连接器尚未使用 MariaDB 进行测试,但来自社区的多份报告表明该连接器已成功用于该数据库。MariaDB 的官方支持计划在未来的 Debezium 版本中使用。
支持的 MySQL 布署
PS D:\kingbaseProject\crmp-data-syncjob> mvn dependency:tree -Dincludes="io.debezium" -Dverbose [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] 多数据源同步服务 [pom] [INFO] crmp-data-syncjob-common [jar] [INFO] crmp-data-syncjob-domain [jar] [INFO] crmp-data-syncjob-dao [jar] [INFO] crmp-data-syncjob-service [jar] [INFO] crmp-data-syncjob-web [jar] [INFO] [INFO] -------------------< com.crmp.ecc:crmp-data-syncjob >------------------- [INFO] Building 多数据源同步服务 1.0.0-SNAPSHOT [1/6] [INFO] from pom.xml [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob --- [INFO] com.crmp.ecc:crmp-data-syncjob:pom:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-common >---------------- [INFO] Building crmp-data-syncjob-common 1.0.0-SNAPSHOT [2/6] [INFO] from crmp-data-syncjob-common\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-common --- [INFO] com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-domain >---------------- [INFO] Building crmp-data-syncjob-domain 1.0.0-SNAPSHOT [3/6] [INFO] from crmp-data-syncjob-domain\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-domain --- [INFO] com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-dao >----------------- [INFO] Building crmp-data-syncjob-dao 1.0.0-SNAPSHOT [4/6] [INFO] from crmp-data-syncjob-dao\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-dao --- [INFO] com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-service >--------------- [INFO] Building crmp-data-syncjob-service 1.0.0-SNAPSHOT [5/6] [INFO] from crmp-data-syncjob-service\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-service --- [INFO] com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-web >----------------- [INFO] Building crmp-data-syncjob-web 1.0.0-SNAPSHOT [6/6] [INFO] from crmp-data-syncjob-web\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-web --- [INFO] com.crmp.ecc:crmp-data-syncjob-web:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT:compile [INFO] | +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for 多数据源同步服务 1.0.0-SNAPSHOT: [INFO] [INFO] 多数据源同步服务 ........................................... SUCCESS [ 3.154 s] [INFO] crmp-data-syncjob-common ........................... SUCCESS [ 0.293 s] [INFO] crmp-data-syncjob-domain ........................... SUCCESS [ 0.271 s] [INFO] crmp-data-syncjob-dao .............................. SUCCESS [ 0.241 s] [INFO] crmp-data-syncjob-service .......................... SUCCESS [ 0.244 s] [INFO] crmp-data-syncjob-web .............................. SUCCESS [ 0.247 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.944 s [INFO] Finished at: 2025-11-25T11:43:19+08:00 [INFO] ------------------------------------------------------------------------ 请仔细检查是否还存在依赖错误!
11-26
<think> 我们之前已经通过`mvn dependency:tree -Dincludes="io.debezium" -Dverbose`命令查看了项目中的所有debezium依赖,结果显示所有debezium依赖的版本都是1.9.8.Final,没有出现其他版本。因此,在项目依赖树中,不存在debezium的版本冲突。 但是,用户提到了一个报错:“Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory”。这个类属于`debezium-storage-jdbc`模块,在debezium 1.9.8.Final中确实存在。因此,这个错误可能是因为缺少这个依赖导致的。 所以,我们需要检查项目是否引入了`debezium-storage-jdbc`依赖。如果没有引入,则需要添加;如果引入了,但版本不对,则需要调整版本。 ### 解决方案 1. **检查依赖树中是否有`debezium-storage-jdbc`模块** 执行命令: ```powershell mvn dependency:tree -Dincludes="io.debezium:debezium-storage-jdbc" ``` 2. **如果没有找到,则在项目中添加依赖** 在需要使用JdbcSchemaHistory的模块中(可能是主项目或某个子模块),添加以下依赖: ```xml <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-storage-jdbc</artifactId> <version>1.9.8.Final</version> </dependency> ``` 3. **如果已经存在,检查版本是否一致** 确保版本与其他debezium依赖一致(1.9.8.Final)。 ### 验证步骤 1. 添加依赖后,重新运行`mvn dependency:tree -Dincludes="io.debezium"`,确认`debezium-storage-jdbc`的版本是1.9.8.Final。 2. 重新构建项目,看是否解决类找不到的问题。 ### 原因分析 - `JdbcSchemaHistory`类在`debezium-storage-jdbc`模块中,该模块不是`debezium-core`或`debezium-connector-mysql`等模块的一部分,所以需要单独引入。 -Debezium 1.9版本中,历史存储被模块化,因此使用JDBC存储历史记录需要显式添加这个依赖。 ### 注意事项 - 确保所有debezium相关依赖版本一致,避免因版本不一致导致类加载问题。 ### 相关问题 1. 如何统一管理Maven项目中多个模块的依赖版本? 2. 除了JDBC存储,Debezium还支持哪些历史记录存储方式? 3. 在Flink CDC中使用Debezium时,如何配置历史记录存储? 4. 如何验证`debezium-storage-jdbc`依赖是否正确添加到类路径? 请根据上述步骤操作,如果问题仍然存在,请检查类路径中是否确实包含了该jar包。 引用参考: [^1]: Debezium报错处理系列之七十七:Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory。该错误通常是因为缺少`debezium-storage-jdbc`依赖。 </think>基于之前的依赖树分析和报错信息,以下是关于 Debezium 依赖错误的系统分析和解决方案: ### 🔍 依赖错误诊断 1. **类缺失问题分析** `Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory` 错误表明: - 所需类属于 `debezium-storage-jdbc` 模块 - 当前依赖树中**未包含此模块**(仅存在 `core`/`connector` 等基础模块) - 根本原因:缺少关键依赖项 2. **依赖完整性验证** 数学表达: $$ \text{Required Modules} = \{\text{core}, \text{api}, \text{connector}, \color{red}{\text{storage-jdbc}} \} $$ 当前项目: $$ \text{Actual Modules} = \{\text{core}, \text{api}, \text{connector}\} \implies \text{storage-jdbc} \notin \text{Dependency Set} $$ --- ### 🛠️ 解决方案 #### 步骤1:添加缺失依赖 在父 POM 或使用 CDC 的模块中添加: ```xml <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-storage-jdbc</artifactId> <version>1.9.8.Final</version> <!-- 保持版本一致 --> </dependency> ``` #### 步骤2:验证依赖树 执行命令确认依赖引入: ```powershell mvn dependency:tree -Dincludes="io.debezium:debezium-storage-jdbc" ``` 预期输出应包含: ``` [INFO] \- io.debezium:debezium-storage-jdbc:jar:1.9.8.Final:compile ``` #### 步骤3:排除潜在冲突(如需要) 若出现其他版本冲突: ```xml <exclusions> <exclusion> <groupId>org.infinispan</groupId> <!-- 示例:排除冲突组件 --> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> ``` --- ### ⚠️ 关键注意事项 1. **版本一致性** 所有 Debezium 组件必须保持相同版本(当前项目统一使用 `1.9.8.Final`) ```mermaid graph LR A[debezium-core] -->|1.9.8.Final| B[storage-jdbc] C[connector-mysql] -->|1.9.8.Final| B ``` 2. **依赖作用域** `debezium-storage-jdbc` 应设为 `compile` 作用域,确保打包时包含 3. **历史记录配置** 在 CDC 配置中需显式指定: ```properties database.history=io.debezium.storage.jdbc.history.JdbcSchemaHistory database.history.jdbc.url=jdbc:postgresql://localhost/history ``` --- ### 📚 相关知识拓展 1. **Debezium 存储模块关系** ```mermaid classDiagram DebeziumCore --> JdbcSchemaHistory JdbcSchemaHistory : +start() JdbcSchemaHistory : +storeRecord() class JdbcSchemaHistory{ -dataSource +configure() } ``` 2. **常见依赖冲突场景** | 冲突组件 | 症状 | 解决方案 | |---------|------|---------| | Jackson | JSON解析异常 | 统一版本到 `2.12.5` | | Kafka Client | 连接超时 | 保持 `3.2.0` 以上 | | Infinispan | 类加载错误 | 排除旧版本 | --- ### ❓ 相关问题 1. 如何验证 `JdbcSchemaHistory` 是否成功初始化? 2. Debezium 支持哪些历史记录存储方案(除 JDBC 外)? 3. 当遇到 `ClassNotFoundException` 时,如何快速定位缺失依赖? 4. 在多模块项目中如何统一管理 Debezium 子模块版本? > 添加依赖后,请运行测试用例验证历史记录存储功能。若仍有问题,建议检查数据库连接配置[^1][^3]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

onemy

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值