20211123-CanalParseException: column size is not match for table:test

本文介绍了当MySQL表字段增减时,Canal Server中监听实例如何处理此类错误,并提供了两种解决方案:一是通过调整canal-instance配置跳过旧数据并进行全量同步;二是配置忽略DDL更新,但同样需要后续全量同步。重点解析了源码中定位binlog起始位置的逻辑。

canal git源码地址:
https://github.com/alibaba/canal
https://github.com/alibaba/canal/wiki/AdminGuide

场景:当一个mysql表的字段有增减时,canal-server集群下的对应监听的canal-instance则会触发此错误;
2021-11-23 09:29:18.950 [destination = test_instance_lw , address = svc.cluster/127.0.0.1:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:test_instance_lw[com.alibaba.otter.canal.parse.exception.CanalParseException: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: parse row data failed.
Caused by: com.alibaba.otter.canal.parse.exception.CanalParseException: column size is not match for table:test.t_shop,14 vs 13

解决方案(一):
1,配置canal-instance文件直接移动位点至最新的timestamp,但是会丢弃掉之前的binlog更新,所以需要做一次全量的同步更新;
canal.instance.master.timestamp=1636712400000
2,注意要先停止instance服务,canal-client最好也停下,删除zk上记录binlog位点的/otter/canal/destinations/test_instance_lw/1001/cursor

解决方案(二):
配置忽略ddl更新,但是与方案相似,仍然需要再做一次全量;
canal.instance.filter.query.ddl = false

canal源码:
//com.alibaba.otter.canal.parse.inbound.mysql.MysqlEventParser#findStartPositionInternal

protected EntryPosition findStartPositionInternal(ErosaConnection connection) {
        MysqlConnection mysqlConnection = (MysqlConnection) connection;
        LogPosition logPosition = logPositionManager.getLatestIndexBy(destination);
        if (logPosition == null) {// 找不到历史成功记录
            EntryPosition entryPosition = null;
            if (masterInfo != null && mysqlConnection.getConnector().getAddress().equals(masterInfo.getAddress())) {
                entryPosition = masterPosition;
            } else if (standbyInfo != null
                       && mysqlConnection.getConnector().getAddress().equals(standbyInfo.getAddress())) {
                entryPosition = standbyPosition;
            }

            if (entryPosition == null) {
                entryPosition = findEndPositionWithMasterIdAndTimestamp(mysqlConnection); // 默认从当前最后一个位置进行消费
            }

            // 判断一下是否需要按时间订阅
            if (StringUtils.isEmpty(entryPosition.getJournalName())) {
                // 如果没有指定binlogName,尝试按照timestamp进行查找
                if (entryPosition.getTimestamp() != null && entryPosition.getTimestamp() > 0L) {
                    logger.warn("prepare to find start position {}:{}:{}",
                        new Object[] { "", "", entryPosition.getTimestamp() });
                    return findByStartTimeStamp(mysqlConnection, entryPosition.getTimestamp());
                } else {
                    logger.warn("prepare to find start position just show master status");
                    return findEndPositionWithMasterIdAndTimestamp(mysqlConnection); // 默认从当前最后一个位置进行消费
                }
            } else {
                if (entryPosition.getPosition() != null && entryPosition.getPosition() > 0L) {
                    // 如果指定binlogName + offest,直接返回
                    entryPosition = findPositionWithMasterIdAndTimestamp(mysqlConnection, entryPosition);
                    logger.warn("prepare to find start position {}:{}:{}",
                        new Object[] { entryPosition.getJournalName(), entryPosition.getPosition(),
                                entryPosition.getTimestamp() });
                    return entryPosition;
                } else {
                    EntryPosition specificLogFilePosition = null;
                    if (entryPosition.getTimestamp() != null && entryPosition.getTimestamp() > 0L) {
                        // 如果指定binlogName +
                        // timestamp,但没有指定对应的offest,尝试根据时间找一下offest
                        EntryPosition endPosition = findEndPosition(mysqlConnection);
                        if (endPosition != null) {
                            logger.warn("prepare to find start position {}:{}:{}",
                                new Object[] { entryPosition.getJournalName(), "", entryPosition.getTimestamp() });
                            specificLogFilePosition = findAsPerTimestampInSpecificLogFile(mysqlConnection,
                                entryPosition.getTimestamp(),
                                endPosition,
                                entryPosition.getJournalName(),
                                true);
                        }
                    }

                    if (specificLogFilePosition == null) {
                        if (isRdsOssMode()) {
                            // 如果binlog位点不存在,并且属于timestamp不为空,可以返回null走到oss binlog处理
                            return null;
                        }
                        // position不存在,从文件头开始
                        entryPosition.setPosition(BINLOG_START_OFFEST);
                        return entryPosition;
                    } else {
                        return specificLogFilePosition;
                    }
                }
            }
        } else {
            if (logPosition.getIdentity().getSourceAddress().equals(mysqlConnection.getConnector().getAddress())) {
                if (dumpErrorCountThreshold >= 0 && dumpErrorCount > dumpErrorCountThreshold) {
                    // binlog定位位点失败,可能有两个原因:
                    // 1. binlog位点被删除
                    // 2.vip模式的mysql,发生了主备切换,判断一下serverId是否变化,针对这种模式可以发起一次基于时间戳查找合适的binlog位点
                    boolean case2 = (standbyInfo == null || standbyInfo.getAddress() == null)
                                    && logPosition.getPostion().getServerId() != null
                                    && !logPosition.getPostion().getServerId().equals(findServerId(mysqlConnection));
                    if (case2) {
                        EntryPosition findPosition = fallbackFindByStartTimestamp(logPosition, mysqlConnection);
                        dumpErrorCount = 0;
                        return findPosition;
                    }
                    // 处理 binlog 位点被删除的情况,提供自动重置到当前位点的功能
                    // 应用场景: 测试环境不稳定,位点经常被删。强烈不建议在正式环境中开启此控制参数,因为binlog
                    // 丢失调到最新位点也即意味着数据丢失
                    if (isAutoResetLatestPosMode()) {
                        dumpErrorCount = 0;
                        return findEndPosition(mysqlConnection);
                    }
                    Long timestamp = logPosition.getPostion().getTimestamp();
                    if (isRdsOssMode() && (timestamp != null && timestamp > 0)) {
                        // 如果binlog位点不存在,并且属于timestamp不为空,可以返回null走到oss binlog处理
                        return null;
                    }
                } else if (StringUtils.isBlank(logPosition.getPostion().getJournalName())
                           && logPosition.getPostion().getPosition() <= 0
                           && logPosition.getPostion().getTimestamp() > 0) {
                    return fallbackFindByStartTimestamp(logPosition, mysqlConnection);
                }
                // 其余情况
                logger.warn("prepare to find start position just last position\n {}",
                    JsonUtils.marshalToString(logPosition));
                return logPosition.getPostion();
            } else {
                // 针对切换的情况,考虑回退时间
                long newStartTimestamp = logPosition.getPostion().getTimestamp() - fallbackIntervalInSeconds * 1000;
                logger.warn("prepare to find start position by switch {}:{}:{}", new Object[] { "", "",
                        logPosition.getPostion().getTimestamp() });
                return findByStartTimeStamp(mysqlConnection, newStartTimestamp);
            }
        }
    }
解析一下日志:2025-12-04 14:00:48[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[__init__.py:146] start to init com component for parsed data. 2025-12-04 14:00:51[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[__init__.py:188] 当前目标excel文档已打开. 2025-12-04 14:00:51[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[__init__.py:124] start to adjust excel office ui position. 2025-12-04 14:00:51[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[__init__.py:223] start to adjust parsed front UI. 2025-12-04 14:00:51[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[__init__.py:231] adjust parsed front UI succeed. 2025-12-04 14:00:53[MainProcess-13872:Thread-76 (process_request_thread)-13252]INFO[_internal.py:97] 127.0.0.1 - - [04/Dec/2025 14:00:53] "GET /api/v1.0.0/parsed_data?filename=E:/test/MCUCONST/before/MET-M_MCUCONST-CSTD-A0-020-A-XX-XXXX-X.xlsx&win_title=変更前解析結果-E:/test/MCUCONST/before/MET-M_MCUCONST-CSTD-A0-020-A-XX-XXXX-X.xlsx HTTP/1.1" 200 - 2025-12-04 14:00:54[MainProcess-13872:Thread-77 (process_request_thread)-19140]INFO[_internal.py:97] 127.0.0.1 - - [04/Dec/2025 14:00:54] "GET /api/v1.0.0/operate/excel?sheetname=変更履歴(車担用)&coord=C7 HTTP/1.1" 200 - 2025-12-04 14:01:02[MainProcess-13872:Thread-78 (process_request_thread)-13040]INFO[_internal.py:97] 127.0.0.1 - - [04/Dec/2025 14:01:02] "GET /api/v1.0.0/operate/excel?sheetname=MCUCONST&coord=A1 HTTP/1.1" 200 - 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[routes.py:200] confirm parsed received params: file_range: all; content_range: all; form_range: all; chapter_range: all; 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[match_rule.py:56] column_names:['rule_id', 'rule_name', 'rule_type', 'priority', 'module_name', 'data_type', 'rule_statement', 'file_name', 'block_name', 'chapter', 'content', 'data', 'filter', 'action'] 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[match_rule.py:208] Match the rule! 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[match_rule.py:215] Its default rule,add data 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[match_rule.py:249] modify success 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[routes.py:224] web server write 'table_parse_multi_line_header' succeed. 2025-12-04 14:02:34[MainProcess-13872:Thread-80 (process_request_thread)-19072]INFO[_internal.py:97] 127.0.0.1 - - [04/Dec/2025 14:02:34] "POST /api/v1.0.0/confirm/parsed HTTP/1.1" 200 - 2025-12-04 14:02:37[MainProcess-13872:Thread-81 (process_request_thread)-6260]INFO[__init__.py:359] start to close E:/test/MCUCONST/before/MET-M_MCUCONST-CSTD-A0-020-A-XX-XXXX-X.xlsx excel workbook. 2025-12-04 14:02:37[MainProcess-13872:Thread-81 (process_request_thread)-6260]INFO[__init__.py:371] close E:/test/MCUCONST/before/MET-M_MCUCONST-CSTD-A0-020-A-XX-XXXX-X.xlsx excel workbook succeed. 2025-12-04 14:02:38[MainProcess-13872:Thread-81 (process_request_thread)-6260]INFO[_internal.py:97] 127.0.0.1 - - [04/Dec/2025 14:02:38] "POST /api/v1.0.0/close/office HTTP/1.1" 200 -
最新发布
12-05
内容过长自动换行, <xsl:template match=“TableReport”> <fo:block > <fo:table text-align="left" border-collapse="collapse" letter-spacing="1px" width="100%" border-width="1pt" font-size="2.6mm"> <fo:table-column border-width="0pt" column-width="0.7cm"/> <fo:table-column border-width="0pt" column-width="0.8cm"/> <fo:table-column border-width="0pt" column-width="2.5cm"/> <fo:table-column border-width="0pt" column-width="1.5cm"/> <fo:table-column border-width="0pt" column-width="1cm"/> <fo:table-column border-width="0pt" column-width="1.6cm"/> <fo:table-column border-width="0pt" column-width="1.6cm"/> <fo:table-column border-width="0pt" column-width="3cm"/> <fo:table-column border-width="0pt" column-width="4.7cm"/> <fo:table-column border-width="0pt" column-width="0.8cm"/> <fo:table-column border-width="0pt" column-width="1.1cm"/> <fo:table-header> <fo:table-row height="5mm"> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block text-align="center">序号</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block text-align="center">组织</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block text-align="center">申请单号</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block text-align="center">类型</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block text-align="center">状态</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">工号</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">姓名</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">物料编码</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">物料描述</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">销数</fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block text-align="center">原数量</fo:block> </fo:table-cell> </fo:table-row> </fo:table-header> <!--表明细--> <fo:table-body> <xsl:apply-templates select="TableRow"/> </fo:table-body> </fo:table> </fo:block> <fo:block id="end"/> </xsl:template> <xsl:template match="TableRow"> <fo:table-row height="5mm"> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block><xsl:value-of select="no"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block><xsl:value-of select="orgCode"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style" font-size="2.2mm"> <fo:block><xsl:value-of select="requestNo"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block><xsl:value-of select="noteType"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-left-style"> <fo:block><xsl:value-of select="state"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block><xsl:value-of select="empNo"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style" font-size="2.2mm"> <fo:block><xsl:value-of select="empName"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style" font-size="2.2mm"> <fo:block><xsl:value-of select="itemCode"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style" font-size="2.2mm"> <fo:block wrap-option="wrap" language="zh"><xsl:value-of select="itemDesc"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style" font-size="2.2mm"> <fo:block wrap-option="wrap" language="zh"><xsl:value-of select="qty"/></fo:block> </fo:table-cell> <fo:table-cell xsl:use-attribute-sets="border-cell-style"> <fo:block><xsl:value-of select="holdQty"/></fo:block> </fo:table-cell> </fo:table-row> </xsl:template>
08-02
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值