记录一次datax同步数据到doris错误[DATA_QUALITY_ERROR]Encountered unqualified data

原创

已于 2024-04-10 15:15:16 修改 · 4k 阅读

15 ·

CC 4.0 BY-SA版权

文章标签：

#python #linux #数据库 #java

于 2024-04-10 15:13:51 首次发布

在使用DataX从MySQL同步数据到Doris时，遇到DATA_QUALITY_ERROR错误，提示遇到了不合格的数据导致同步失败。错误信息中没有提供详细的错误链接，经过排查发现是字段长度超出限制导致的问题。通过逐个字段测试，最终定位并解决了问题。

使用data同步数据，mysql到doris数据库，遇到过没有详细报错的情况，之前一般遇到字段长度或类型转换错误它是有详细错误的一个链接呈现错误原因的，找了很久没找到原因。

错误提示：

[2024-04-10 14:51:38.057][ERROR] >>>> 运行scheduler 模式[standalone]出错.

com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.io.IOException: Failed to flush data to doris.

{"Status":"Fail","Comment":"","BeginTxnTimeMs":1,"Message":"[CANCELLED][DATA_QUALITY_ERROR]Encountered unqualified data, stop processing","NumberUnselectedRows":0,"CommitAndPublishTimeMs":0,"Label":"datax_doris_writer_580a53cc-7fe0-45cc-9142-acac312cdcbc_0","LoadBytes":2455041,"StreamLoadPutTimeMs":6,"NumberTotalRows":0,"WriteDataTimeMs":310,"TxnId":2038723,"LoadTimeMs":319,"TwoPhaseCommit":"false","ReadDataTimeMs":10,"NumberLoadedRows":0,"NumberFilteredRows":0}

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterEmitter.doStreamLoad(DorisWriterEmitter.java:114)

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.flush(DorisWriter.java:123)

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:113)

at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56)

at java.base/java.lang.Thread.run(Thread.java:834)

- java.io.IOException: Failed to flush data to doris.

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterEmitter.doStreamLoad(DorisWriterEmitter.java:114)

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.flush(DorisWriter.java:123)

at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.startWrite(DorisWriter.java:113)

at com.alibaba