hive 在kettle 作业return code 2 的问题 Unexpected exception: Unexpected exception: Unexpected exception:

最新推荐文章于 2024-01-30 16:23:22 发布

原创

最新推荐文章于 2024-01-30 16:23:22 发布 · 702 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hive #kettle

在使用Hive进行数据处理时遇到return code 2的错误，这可能由于向量化执行启用且数据量过大导致。为解决此问题，可以关闭向量化执行模式，通过设置`set hive.vectorized.execution.enabled=false`。日志显示了详细的错误堆栈，主要涉及MapReduce任务的MapOperator和MapJoinOperator。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天突然进去跑不了数据老是报错return code 2 .之前可以跑通的。

日志如下

矢量查询(Vectorized query) 每次处理数据时会将1024行数据组成一个batch进行处理，而不是一行一行进行处理，这样能够显著提高执行速度。
可以通过设置
set hive.vectorized.execution.enabled = true; 但是如果数据量很大，她的batch就会很多，有可能导致return code -2

解决方法：
set hive.vectorized.execution.enabled=false; 控制是否启用查询执行的向量模式。

Log Upload Time: Thu Dec 20 15:38:52 +0800 2018

Log Length: 2999805

Showing 1000000 bytes of 2999805 total. Click
here
for the full log.

   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row 、
   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
   ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: parquet.column.values.dictionary.DictionaryValuesWriter$PlainBinaryDictionaryValuesWriter
   at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:318)
   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
   at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:97)
   at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
   ... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: parquet.column.values.dictionary.DictionaryValuesWriter$PlainBinaryDictionaryValuesWriter
   at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:318)
   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:638)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
   at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:306)
   ... 13 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: Unexpected exception: Unexpected exception: Unexpected exception: parquet.column.values.dictionary.DictionaryValuesWriter$PlainBinaryDictionaryValuesWriter
   at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:318)
   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:638)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.createForwardJoinObject(CommonJoinOperator.java:433)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:525)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:522)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genJoinObject(CommonJoinOperator.java:451)
   at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoi