hive 动态分区(Dynamic Partition)异常处理

最新推荐文章于 2024-08-23 10:11:17 发布

置顶 Data_IT_Farmer

最新推荐文章于 2024-08-23 10:11:17 发布

阅读量5.5k

点赞数 1

分类专栏： hive 动态分区插入文章标签： hive 动态分区插入

hive 同时被 2 个专栏收录

49 篇文章

订阅专栏

动态分区插入

2 篇文章

订阅专栏

本文介绍了Hive在创建动态分区时可能遇到的错误及其原因，当尝试创建的动态分区数量超过默认限制（每个节点100个，总共1000个）时，Hive会抛出异常。文章提供了调整配置以增加最大动态分区数的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

hive 动态分区(Dynamic Partition)异常处理

Changing Hive Dynamic Partition Limits

Symptoms:

Hive enforces limits on the number of dynamic partitions that it creates. The default is 100 dynamic partitions per node, with a total (default) limit of 1000 dynamic partitions across all nodes. However, this can be tuned. If your job tries to create too many dynamic partitions, you may see a stack trace similar to the following:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101

	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:704)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
	at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
	... 18 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1508385188096_13199_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

Cause:

Your Hive job tried to create too many dynamic partitions on a node, exceeding the current limits. The number of dynamic partitions is controlled by the settings forhive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. The default is 100 dynamic partitions on a single node, with a limit of 1000 overall.

Solution:

Set the values for hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode to higher values. For example:

SET hive.exec.dynamic.partition=true;
SET hive.exec.max.dynamic.partitions=2048;
SET hive.exec.max.dynamic.partitions.pernode=256;

参考网址：https://blog.youkuaiyun.com/odailidong/article/details/49884571