spark 1.6 idea本地运行错误

最新推荐文章于 2022-08-23 16:16:34 发布

weixin_30788239

最新推荐文章于 2022-08-23 16:16:34 发布

阅读量200

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 scala 开发工具

原文链接：http://www.cnblogs.com/zhoudayang/p/5260366.html

更新Spark至1.6后，运行程序出现内存不足错误。通过调整JVM参数，如-Xms128m, -Xmx512m等，成功解决了内存限制问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

将spark maven更新至1.6，运行原有程序，发现以下错误：

java.lang.IllegalArgumentException: System memory 259522560 must be at least 4.718592E8. Please use a larger heap size.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:193)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:175)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:354)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:288)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
    at final_paper_time.main(final_paper_time.java:153)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
16/03/10 00:35:36 INFO SparkContext: Successfully stopped SparkContext