Hudi - Could not create payload for class

原创

已于 2023-05-18 16:15:07 修改 · 645 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #hudi #flink

于 2023-05-18 16:13:02 首次发布

设置错误的 payload：

set `hoodie.datasource.write.payload.class`=`org.apache.hudi.common.model.PartialUpdateAvroPayloadX`;

在执行 insert 时报错：

2023-05-18 15:50:23 An error occurred while calling o42161071.execute.
: org.apache.calcite.avatica.AvaticaSqlException: Error -1 (00000) : Error while executing SQL: Remote driver error: CalciteSQLException: Failed to fetch query result (code=2002) -> RuntimeException: While executing SQL [INSERT INTO `test_db`.`t31` partition (ds) (`_hoodie_commit_time`, `_hoodie_commit_seqno`, `_hoodie_record_key`, `_hoodie_partition_path`, `_hoodie_file_name`, `ut`, `pk`, `f0`, `f1`, `f2`, `f3`, `f4`, `ds`)
(SELECT CAST(NULL AS STRING) `_hoodie_commit_time`, CAST(NULL AS STRING) `_hoodie_commit_seqno`, CAST(NULL AS STRING) `_hoodie_record_key`, CAST(NULL AS STRING) `_hoodie_partition_path`, CAST(NULL AS STRING) `_hoodie_file_name`, CURRENT_TIMESTAMP `ut`, 1006 `pk`, 1 `f0`, CAST(NULL AS BIGINT) `f1`, CAST(NULL AS BIGINT) `f2`, CAST(NULL AS BIGINT) `f3`, CAST(NULL AS BIGINT) `f4`, 20230101 `ds`)] on JDBC sub-schema -> SQLException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20230518155010549
	at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:75)
	at org.apache.hudi.table.action.deltacommit.SparkUpsertDeltaCommitActionExecutor.execute(SparkUpsertDeltaCommitActionExecutor.java:45)
	at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsert(HoodieSparkMergeOnReadTable.java:88)
	at org.apache.hudi.table.HoodieSparkMergeOnReadTable.upsert(HoodieSparkMergeOnReadTable.java:80)
	at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:140)
	at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:206)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:363)
	at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableComman