tidb-dm报警DM_sync_process_exists_with_error排查

原创

已于 2022-06-29 17:17:07 修改 · 314 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#mysql #数据库 #服务器

于 2022-06-29 16:40:09 首次发布

这篇博客探讨了一次DM同步任务在遇到DM_sync_process_exists_with_error报警后，如何在一分钟后自动恢复的问题。日志显示，错误源于DM-worker的invalidconnection，进一步分析发现与TiDB的waitresponseiscancelled和Tikv的锁及backoff有关。PD的日志揭示了hot-read-leader调度可能导致的backoff，而业务SQL可能是锁冲突的源头。解决方案可能涉及优化SQL和调整集群配置。

一、背景

dm同步任务报警DM_sync_process_exists_with_error，一分钟后自动恢复，想着排查一下原因

二、观测日志报错

1.dm日志

[2022/06/28 14:31:13.364 +00:00] [ERROR] [db.go:201] ["execute statements failed after retry"] [task=task-name] [unit="binlog replication"] [queries="[sql]"] [arguments="[[]]"] [error="[code=10006:class=database:scope=not-set:level=high], Message: execute statement failed: commit, RawCause: invalid connection"]

2.上游mysql日志

2022-06-28T14:31:19.413211Z 28801 [Note] Aborted connection 28801 to db: 'unconnected' user: '***' host: 'ip' (Got an error reading communication packets)
2022-06-28T14:31:22.154980Z 28802 [Note] Aborted connection 28802 to db: 'unconnected' user: '***' host: 'ip' (Got an error reading communication packets)
2022-06-28T14:31:32.158508Z 28804 [Note] Start binlog_dump to master_thread_id(28804) slave_server(429505412), pos(mysql-bin-changelog.103037, 36247149)
2022-06-28T14:31:32.158739Z 28803 [Note] Start binlog_dump to master_thread_id(28803) slave_server(429505202), pos(mysql-bin-changelog.103037, 40373779)

3.下游tidb日志

[2022/06/28 14:31:12.419 +00:00] [WARN] [client_batch.go:638] ["wait resp

最低0.47元/天解锁文章