New Memory allocation 1046759 bytes is smaller than the minimum allocation size of 1048576 bytes.

本文链接：https://blog.youkuaiyun.com/xiaolinzi007/article/details/70174829

在执行Spark作业时遇到了org.apache.spark.SparkException，任务因stage失败而中止，错误信息指出内存分配1046759字节小于最小分配1048576字节。此问题与Hive的Parquet格式和内存管理有关，可以通过调整Hive配置`parquet.memory.min.chunk.size`和`hive.exec.max.dynamic.partitions`来解决，分别设置为100000和100000。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 78, hdp57.car.bj2.yongche.com): org.apache.hadoop.hive.ql.metadata.HiveException:
 parquet.hadoop.MemoryManager$1: New Memory allocation 1046759 bytes is smaller than the minimum allocation size of 1048576 bytes.

at org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer.org$apache$spark$sql$hive$SparkHiveDynamicPartitionWriterContainer$$newWriter$1(hiveWriterContainers.scala:240)

at org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer$$anonfun$getLocalFileWriter$1.apply(hiveWriterContainers.scala:249)

at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1$1.apply(InsertIntoHiveTable.scala:112)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1$1.apply(InsertIntoHiveTable.scala:104)

at scala.collection.Iterator$class.foreach(Iterator.scala:727)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.org$apache$spark$sql$hive$execution$InsertIntoHiveTable$$writeToFile$1(InsertIntoHiveTable.scala:104)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.apply(InsertIntoHiveTable.scala:84)

at org.apache.spark.scheduler.Task.run(Task.scala:89)

解决方法:

hiveContext.setConf("parquet.memory.min.chunk.size", "100000")

hiveContext.setConf("hive.exec.max.dynamic.partitions", "100000")