Spark-submit:System memory 466092032 must be at least 471859200

最新推荐文章于 2024-02-17 19:11:27 发布

原创

最新推荐文章于 2024-02-17 19:11:27 发布 · 1.9k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#spark #System memory

在运行Standalone-client模式时遇到如下错误：

Spark Executor Command: "/usr/local/jdk1.8.0_181/bin/java" "-cp" "/usr/local/spark-2.3.0/conf/:/usr/local/spark-2.3.0/jars/*:/usr/local/hadoop-2.7.6/etc/hadoop/" "-Xmx500M" "-Dspark.driver.port=43614" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@hpmaster:43614" "--executor-id" "32" "--hostname" "192.168.199.212" "--cores" "1" "--app-id" "app-20190212233950-0001" "--worker-url" "spark://Worker@192.168.199.212:7079"
========================================

Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
at org.apache.s