Debezium指定binlog位置初始化异常排查修复

最新推荐文章于 2025-11-17 20:26:14 发布

原创

最新推荐文章于 2025-11-17 20:26:14 发布 · 4.3k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#mysql #debezium #binlog #offset #指定位置

本文详细介绍了在使用Debezium与Flink集成过程中遇到的binlog读取异常问题。当配置FileDatabaseHistory时，系统无法识别schema导致错误。通过日志分析，发现在schema_only_recovery模式下，FileDatabaseHistory缺少必要的初始化步骤，而FlinkDatabaseHistory则能正确执行。问题根源在于snapshottingTask.shouldSkipSnapshot()的逻辑，进一步发现FileDatabaseHistory的schema初始化未执行。最终，通过对比和调试，提出了修复方案，即修改exists方法的判断条件，确保正确执行初始化。

异常现象

最近项目需要使用Debezium指定binlog位置读取数据，当配置FileDatabaseHistory 保存offset就会出现无法识别schema异常。

14:52:18.237 [blc-9.135.12.10:3307] ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered change event 'Event{header=EventHeaderV4{timestamp=1652710509000, eventType=TABLE_MAP, serverId=123454, headerLength=19, dataLength=30, nextPosition=5761, flags=0}, data=TableMapEventData{tableId=375, database='etl', table='test', columnTypes=3, 15, columnMetadata=0, 64, columnNullability={0, 1}, eventMetadata=null}}' at offset {
   
   transaction_id=null, file=mysql-bin.000001, pos=5641, server_id=123454, event=1} for table etl.test whose schema isn't known to this connector. One possible cause is an incomplete database history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=5712 --stop-position=5761 --verbose mysql-bin.000001
14:52:18.237 [blc-9.135.12.10:3307] ERROR io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during binlog processing. Last offset stored = {
   
   transaction_id=null, file=mysql-bin.000001, pos=5641