MySQL-CDC Caused by: java.lang.IllegalStateException: The connector is trying to read binlog问题解决

Warsen Liu

已于 2025-01-21 10:53:21 修改

阅读量566

点赞数 5

CC 4.0 BY-SA版权

文章标签： mysql 数据库

于 2025-01-21 10:42:26 首次发布

本文链接：https://blog.youkuaiyun.com/Junyaxx/article/details/145277547

总结：MySQL-router连接随机某个节点导致此问题，直连固定节点即可

没有使用负载均衡连接所有节点可以跳过此文章，推荐方案链接

报错如下：

java.lang.RuntimeException: One or more fetchers have encountered exception
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:147)
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:160)
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:93)
	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:111)
	at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156)
	at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:116)
	at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
	at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:121)
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:703)
	at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1004)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165)
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:81)
	... 5 more
Caused by: java.lang.IllegalStateException: The connector is trying to read binlog starting at Struct{version=1.6.4.Final,connector=mysql,name=mysql_binlog_source,ts_ms=1737344179327,db=,server_id=0,file=mysql-bin.000856,pos=278802469,row=0}, but this is no longer available on the server. Reconfigure the connector to use a snapshot when needed.
	at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch.MySqlSourceFetchTaskContext.loadStartingOffsetState(MySqlSourceFetchTaskContext.java:266)
	at org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch.MySqlSourceFetchTaskContext.configure(MySqlSourceFetchTaskContext.java:119)
	at org.apache.seatunnel.connectors.cdc.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:97)
	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.checkSplitOrStartNext(IncrementalSourceSplitReader.java:147)
	at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:71)
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.FetchTask.run(FetchTask.java:54)
	at org.apache.seatunnel.connectors.seatunnel.common.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:162)
	... 6 more

看了下现有的问题排查解决方案，排查下来并不能解决我的问题,大部分帖子说的都是binlog文件超过有效期了导致读取报这个错，但是如果设置7天这种超长的时间范围，理论上读取最新的binlog就不存在这种情况，如果还有这种报错，比如启动就抛出这种异常的，可以按我的解决方案试试。

我的问题是使用了mysql-router的连接，不是mysql集群的可以不看了，存在这个问题的原因就是因为router的负载均衡，导致程序初始化时获取的连接查询binlog的最新文件和偏移量为当前连接的mysql节点，但是后续查询数据是池化过的连接，可能连接到别的mysql节点了，而如果你的mysql集群各个节点最新写入的binlog文件名不一致，则存在上述问题，它会导致程序在启动时或运行过程中报这个错，我使用的是seatunnel任务调度，经测试任务配置连接时固定使用其中某一个节点的方式就没出现这个问题了。

可能我说的不够清楚，具体定位方式是直连mysql集群各个节点，使用如下命令：