Debezium实战-第5章 转换

这篇博客深入探讨了Debezium在数据库变更事件处理中的转换功能,包括主题路由、新记录状态提取、MongoDB特定处理、消息过滤和基于内容的路由。内容涵盖了如何配置SMT以有选择地应用转换,确保数据一致性,并展示了如何根据事件内容路由到不同主题。此外,还讨论了如何处理MongoDB的复杂消息结构,以及如何过滤和路由数据变更事件,以满足不同场景的需求。

5 转换

5.1 概览

可以使用转换配置连接器,以使每个消息修改轻量级。Debezium 提供了几个单消息转换(SMT),您可以使用它们在将记录发送到 Apache Kafka(通过将它们应用到 Debezium 连接器)之前修改记录,或者在接收器连接器从 Kafka 读取它们时修改它们。Debezium Server也支持使用 SMT。

Debezium 提供以下 SMT:

转换 描述
主题路由 根据应用于原始主题名称的正则表达式将记录重新路由到不同的主题。
PS D:\kingbaseProject\crmp-data-syncjob> mvn dependency:tree -Dincludes="io.debezium" -Dverbose [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] 多数据源同步服务 [pom] [INFO] crmp-data-syncjob-common [jar] [INFO] crmp-data-syncjob-domain [jar] [INFO] crmp-data-syncjob-dao [jar] [INFO] crmp-data-syncjob-service [jar] [INFO] crmp-data-syncjob-web [jar] [INFO] [INFO] -------------------< com.crmp.ecc:crmp-data-syncjob >------------------- [INFO] Building 多数据源同步服务 1.0.0-SNAPSHOT [1/6] [INFO] from pom.xml [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob --- [INFO] com.crmp.ecc:crmp-data-syncjob:pom:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-common >---------------- [INFO] Building crmp-data-syncjob-common 1.0.0-SNAPSHOT [2/6] [INFO] from crmp-data-syncjob-common\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-common --- [INFO] com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-domain >---------------- [INFO] Building crmp-data-syncjob-domain 1.0.0-SNAPSHOT [3/6] [INFO] from crmp-data-syncjob-domain\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-domain --- [INFO] com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-dao >----------------- [INFO] Building crmp-data-syncjob-dao 1.0.0-SNAPSHOT [4/6] [INFO] from crmp-data-syncjob-dao\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-dao --- [INFO] com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-service >--------------- [INFO] Building crmp-data-syncjob-service 1.0.0-SNAPSHOT [5/6] [INFO] from crmp-data-syncjob-service\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-service --- [INFO] com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-web >----------------- [INFO] Building crmp-data-syncjob-web 1.0.0-SNAPSHOT [6/6] [INFO] from crmp-data-syncjob-web\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-web --- [INFO] com.crmp.ecc:crmp-data-syncjob-web:jar:1.0.0-SNAPSHOT [INFO] +- com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT:compile [INFO] | +- com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT:compile (version managed from 1.0.0-SNAPSHOT) [INFO] | | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | +- (io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- (io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile (version managed from 1.9.8.Final) [INFO] | \- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- (io.debezium:debezium-core:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] \- (io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile - version managed from 1.9.8.Final; omitted for duplicate) [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for 多数据源同步服务 1.0.0-SNAPSHOT: [INFO] [INFO] 多数据源同步服务 ........................................... SUCCESS [ 3.154 s] [INFO] crmp-data-syncjob-common ........................... SUCCESS [ 0.293 s] [INFO] crmp-data-syncjob-domain ........................... SUCCESS [ 0.271 s] [INFO] crmp-data-syncjob-dao .............................. SUCCESS [ 0.241 s] [INFO] crmp-data-syncjob-service .......................... SUCCESS [ 0.244 s] [INFO] crmp-data-syncjob-web .............................. SUCCESS [ 0.247 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.944 s [INFO] Finished at: 2025-11-25T11:43:19+08:00 [INFO] ------------------------------------------------------------------------ 请仔细检查是否还存在依赖错误!
最新发布
11-26
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

onemy

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值