spark dataframe unionall

最新推荐文章于 2023-09-12 06:59:22 发布

weixin_34205826

最新推荐文章于 2023-09-12 06:59:22 发布

阅读量755

点赞数

文章标签：大数据 scala 数据库

本文通过一个具体的例子，介绍了在使用Spark进行DataFrame UnionAll操作时可能遇到的错误及原因。当尝试将不同列数的表进行联合时，会抛出AnalysisException异常。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天本来想写一个spark dataframe unionall的demo,由于粗心报下面错误：

Exception in thread "main" org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the same number of columns, but the left table has 3 columns and the right has 4;
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:38)
at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:170)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:50)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:121)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:50)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)
at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$withPlan(DataFrame.scala:2127)
at org.apache.spark.sql.DataFrame.unionAll(DataFrame.scala:1008)
at com.xiaoju.arch.engine.spark.ReadHiveDemo.main(ReadHiveDemo.java:74)

错误信息很简单：两张表union，至少字段保持一致吧，汗~~~~