运行Spark GraphX Pregel出现Issue communicating with driver in heartbeater异常

原创

于 2020-06-23 15:44:39 发布 · 3.8k 阅读

6 ·

CC 4.0 BY-SA版权

最近基于Spark GraphX Pregel开发了一个ETL任务，运行过程中会报Issue communicating with driver in heartbeater，然后就是Connection refused错误。

经分析，Spark GraphX Pregel在每次迭代都会产生新的job, 如果迭代的次数多，会产生大量的job, driver会被阻塞造成通讯失败，任务中断。

解决办法：调整driver-memory参数。

spark.driver.memory 1g Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file. 1.1.1