Debezium实战-第5章 转换

这篇博客深入探讨了Debezium在数据库变更事件处理中的转换功能,包括主题路由、新记录状态提取、MongoDB特定处理、消息过滤和基于内容的路由。内容涵盖了如何配置SMT以有选择地应用转换,确保数据一致性,并展示了如何根据事件内容路由到不同主题。此外,还讨论了如何处理MongoDB的复杂消息结构,以及如何过滤和路由数据变更事件,以满足不同场景的需求。

5 转换

5.1 概览

可以使用转换配置连接器,以使每个消息修改轻量级。Debezium 提供了几个单消息转换(SMT),您可以使用它们在将记录发送到 Apache Kafka(通过将它们应用到 Debezium 连接器)之前修改记录,或者在接收器连接器从 Kafka 读取它们时修改它们。Debezium Server也支持使用 SMT。

Debezium 提供以下 SMT:

转换 描述
主题路由 根据应用于原始主题名称的正则表达式将记录重新路由到不同的主题。
PS D:\kingbaseProject\crmp-data-syncjob> mvn dependency:tree -Dincludes="io.debezium" -Dverbose [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] 多数据源同步服务 [pom] [INFO] crmp-data-syncjob-common [jar] [INFO] crmp-data-syncjob-domain [jar] [INFO] crmp-data-syncjob-dao [jar] [INFO] crmp-data-syncjob-service [jar] [INFO] crmp-data-syncjob-web [jar] [INFO] [INFO] -------------------< com.crmp.ecc:crmp-data-syncjob >------------------- [INFO] Building 多数据源同步服务 1.0.0-SNAPSHOT [1/6] [INFO] from pom.xml [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob --- [INFO] com.crmp.ecc:crmp-data-syncjob:pom:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-common >---------------- [INFO] Building crmp-data-syncjob-common 1.0.0-SNAPSHOT [2/6] [INFO] from crmp-data-syncjob-common\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-common --- [INFO] com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-domain >---------------- [INFO] Building crmp-data-syncjob-domain 1.0.0-SNAPSHOT [3/6] [INFO] from crmp-data-syncjob-domain\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-domain --- [INFO] com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-dao >----------------- [INFO] Building crmp-data-syncjob-dao 1.0.0-SNAPSHOT [4/6] [INFO] from crmp-data-syncjob-dao\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-dao --- [INFO] com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-service >--------------- [INFO] Building crmp-data-syncjob-service 1.0.0-SNAPSHOT [5/6] [INFO] from crmp-data-syncjob-service\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-service --- [INFO] com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-web >----------------- [INFO] Building crmp-data-syncjob-web 1.0.0-SNAPSHOT [6/6] [INFO] from crmp-data-syncjob-web\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-web --- [INFO] com.crmp.ecc:crmp-data-syncjob-web:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT:compile [INFO] | +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for 多数据源同步服务 1.0.0-SNAPSHOT: [INFO] [INFO] 多数据源同步服务 ........................................... SUCCESS [ 3.154 s] [INFO] crmp-data-syncjob-common ........................... SUCCESS [ 0.293 s] [INFO] crmp-data-syncjob-domain ........................... SUCCESS [ 0.271 s] [INFO] crmp-data-syncjob-dao .............................. SUCCESS [ 0.241 s] [INFO] crmp-data-syncjob-service .......................... SUCCESS [ 0.244 s] [INFO] crmp-data-syncjob-web .............................. SUCCESS [ 0.247 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.944 s [INFO] Finished at: 2025-11-25T11:43:19+08:00 [INFO] ------------------------------------------------------------------------ 请仔细检查是否还存在依赖错误!
最新发布
11-26
<think> 我们之前已经通过`mvn dependency:tree -Dincludes="io.debezium" -Dverbose`命令查看了项目中的所有debezium依赖,结果显示所有debezium依赖的版本都是1.9.8.Final,没有出现其他版本。因此,在项目依赖树中,存在debezium的版本冲突。 但是,用户提到了一个报错:“Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory”。这个类属于`debezium-storage-jdbc`模块,在debezium 1.9.8.Final中确实存在。因此,这个错误可能是因为缺少这个依赖导致的。 所以,我们需要检查项目是否引入了`debezium-storage-jdbc`依赖。如果没有引入,则需要添加;如果引入了,但版本对,则需要调整版本。 ### 解决方案 1. **检查依赖树中是否有`debezium-storage-jdbc`模块** 执行命令: ```powershell mvn dependency:tree -Dincludes="io.debezium:debezium-storage-jdbc" ``` 2. **如果没有找到,则在项目中添加依赖** 在需要使用JdbcSchemaHistory的模块中(可能是主项目或某个子模块),添加以下依赖: ```xml <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-storage-jdbc</artifactId> <version>1.9.8.Final</version> </dependency> ``` 3. **如果已经存在,检查版本是否一致** 确保版本与其他debezium依赖一致(1.9.8.Final)。 ### 验证步骤 1. 添加依赖后,重新运行`mvn dependency:tree -Dincludes="io.debezium"`,确认`debezium-storage-jdbc`的版本是1.9.8.Final。 2. 重新构建项目,看是否解决类找到的问题。 ### 原因分析 - `JdbcSchemaHistory`类在`debezium-storage-jdbc`模块中,该模块是`debezium-core`或`debezium-connector-mysql`等模块的一部分,所以需要单独引入。 -Debezium 1.9版本中,历史存储被模块化,因此使用JDBC存储历史记录需要显式添加这个依赖。 ### 注意事项 - 确保所有debezium相关依赖版本一致,避免因版本一致导致类加载问题。 ### 相关问题 1. 如何统一管理Maven项目中多个模块的依赖版本? 2. 除了JDBC存储,Debezium还支持哪些历史记录存储方式? 3. 在Flink CDC中使用Debezium时,如何配置历史记录存储? 4. 如何验证`debezium-storage-jdbc`依赖是否正确添加到类路径? 请根据上述步骤操作,如果问题仍然存在,请检查类路径中是否确实包含了该jar包。 引用参考: [^1]: Debezium报错处理系列之七十七:Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory。该错误通常是因为缺少`debezium-storage-jdbc`依赖。 </think>基于之前的依赖树分析和报错信息,以下是关于 Debezium 依赖错误的系统分析和解决方案: ### 🔍 依赖错误诊断 1. **类缺失问题分析** `Unable to find class io.debezium.storage.jdbc.history.JdbcSchemaHistory` 错误表明: - 所需类属于 `debezium-storage-jdbc` 模块 - 当前依赖树中**未包含此模块**(仅存在 `core`/`connector` 等基础模块) - 根本原因:缺少关键依赖项 2. **依赖完整性验证** 数学表达: $$ \text{Required Modules} = \{\text{core}, \text{api}, \text{connector}, \color{red}{\text{storage-jdbc}} \} $$ 当前项目: $$ \text{Actual Modules} = \{\text{core}, \text{api}, \text{connector}\} \implies \text{storage-jdbc} \notin \text{Dependency Set} $$ --- ### 🛠️ 解决方案 #### 步骤1:添加缺失依赖 在父 POM 或使用 CDC 的模块中添加: ```xml <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-storage-jdbc</artifactId> <version>1.9.8.Final</version> <!-- 保持版本一致 --> </dependency> ``` #### 步骤2:验证依赖树 执行命令确认依赖引入: ```powershell mvn dependency:tree -Dincludes="io.debezium:debezium-storage-jdbc" ``` 预期输出应包含: ``` [INFO] \- io.debezium:debezium-storage-jdbc:jar:1.9.8.Final:compile ``` #### 步骤3:排除潜在冲突(如需要) 若出现其他版本冲突: ```xml <exclusions> <exclusion> <groupId>org.infinispan</groupId> <!-- 示例:排除冲突组件 --> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> ``` --- ### ⚠️ 关键注意事项 1. **版本一致性** 所有 Debezium 组件必须保持相同版本(当前项目统一使用 `1.9.8.Final`) ```mermaid graph LR A[debezium-core] -->|1.9.8.Final| B[storage-jdbc] C[connector-mysql] -->|1.9.8.Final| B ``` 2. **依赖作用域** `debezium-storage-jdbc` 应设为 `compile` 作用域,确保打包时包含 3. **历史记录配置** 在 CDC 配置中需显式指定: ```properties database.history=io.debezium.storage.jdbc.history.JdbcSchemaHistory database.history.jdbc.url=jdbc:postgresql://localhost/history ``` --- ### 📚 相关知识拓展 1. **Debezium 存储模块关系** ```mermaid classDiagram DebeziumCore --> JdbcSchemaHistory JdbcSchemaHistory : +start() JdbcSchemaHistory : +storeRecord() class JdbcSchemaHistory{ -dataSource +configure() } ``` 2. **常见依赖冲突场景** | 冲突组件 | 症状 | 解决方案 | |---------|------|---------| | Jackson | JSON解析异常 | 统一版本到 `2.12.5` | | Kafka Client | 连接超时 | 保持 `3.2.0` 以上 | | Infinispan | 类加载错误 | 排除旧版本 | --- ### ❓ 相关问题 1. 如何验证 `JdbcSchemaHistory` 是否成功初始化? 2. Debezium 支持哪些历史记录存储方案(除 JDBC 外)? 3. 当遇到 `ClassNotFoundException` 时,如何快速定位缺失依赖? 4. 在多模块项目中如何统一管理 Debezium 子模块版本? > 添加依赖后,请运行测试用例验证历史记录存储功能。若仍有问题,建议检查数据库连接配置[^1][^3]
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

onemy

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值