【error】SparkUI端口被占用

SparkUI端口占用问题及解决方案

最新推荐文章于 2023-07-15 22:55:21 发布

原创最新推荐文章于 2023-07-15 22:55:21 发布 · 1.9k 阅读

2 ·

CC 4.0 BY-SA版权

Spark 同时被 2 个专栏收录

19 篇文章

订阅专栏

Error

15 篇文章

订阅专栏

当Spark任务启动时，SparkUI默认使用4040端口，若被占用会尝试递增端口，最多重试16次。超过次数会导致任务失败。解决办法是在创建SparkConf时设置`spark.port.maxRetries`参数为更高值，如100，以增加重试连接端口的次数。

 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:317)
	at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$newConnector$1(JettyUtils.scala:339)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$httpConnect$1(JettyUtils.scala:366)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:369)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:369)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2234)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2226)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:369)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:460)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:460)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:460)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:839)
	at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:85)
	at flume_push_streaming$.main(flume_push_streaming.scala:11)
	at flume_push_streaming.main(flume_push_streaming.scala)