spark报错：ERROR SparkUI: Failed to bind SparkUI

最新推荐文章于 2024-07-15 17:58:41 发布

原创最新推荐文章于 2024-07-15 17:58:41 发布 · 679 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #scala #大数据

文章描述了在IntelliJIDEA中运行Spark应用时遇到SparkUI启动失败的问题，原因是端口冲突。解决方案是通过增加`spark.port.maxRetries`的值来尝试更多重试，或者显式设置SparkUI的端口到可用的端口。作者通过在SparkSession构建器中添加配置项`.config(spark.port.maxRetries,10000)`成功解决了问题。

kUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.

at org.sparkproject.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)

at org.sparkproject.jetty.server.ServerConnector.open(ServerConnector.java:310)

at org.sparkproject.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)

at org.sparkproject.jetty.server.ServerConnector.doStart(ServerConnector.java:234)

at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.apache.spark.ui.JettyUtils$.newConnector$1(JettyUtils.scala:302)

at org.apache.spark.ui.JettyUtils$.httpConnect$1(JettyUtils.scala:333)

at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5(JettyUtils.scala:336)

at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5$adapted(JettyUtils.scala:336)

at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2331)

at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)

at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2323)

at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:337)

at org.apache.spark.ui.WebUI.bind(WebUI.scala:146)

at org.apache.spark.SparkContext.$anonfun$new$11(SparkContext.scala:486)

at org.apache.spark.SparkContext.$anonfun$new$11$adapted(SparkContext.scala:486)

at scala.Option.foreach(Option.scala:407)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:486)

at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2672)

at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:945)

at scala.Option.getOrElse(Option.scala:189)

at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:939)

at com.mm.rdd.WordCount$.main(WordCount.scala:6)

at com.mm.rdd.WordCount.main(WordCount.scala)

23/05/08 11:46:11 INFO DiskBlockManager: Shutdown hook called

23/05/08 11:46:11 INFO ShutdownHookManager: Shutdown hook called

解决办法：

最终解决方案：

我是在IntellijIdea下运行的，试着安装了windows版本下的hadoop3.2.4核spark3.2.4，都不行；最后是创建统一入口SparkSession，指令后加了一条.config("spark.port.maxRetries",10000)：

val spark = SparkSession.builder().master("local[*]").appName("word count").config("spark.port.maxRetries",10000).getOrCreate()；成功了。

spark报错：ERROR SparkUI: Failed to bind SparkUI

1 条评论