spark报错：ERROR SparkUI: Failed to bind SparkUI，解决办法-优快云博客

当Spark任务因默认端口4040被占用导致无法运行时，SparkUI会尝试16次寻找可用端口。如果失败，可以修改配置，如在启动命令中添加`–conf spark.port.maxRetries=100`，增加尝试次数来解决问题。错误信息显示了由于端口冲突，SparkUI服务启动失败。

错误原因：

每一个Spark任务都会占用一个SparkUI端口，默认为4040，如果被占用则依次递增端口重试。但是有个默认重试次数，为16次。16次重试都失败后，会放弃该任务的运行。
参考：
http://blog.youkuaiyun.com/a921122/article/details/45095845

解决办法

根据提示，可以在命令行通过配置，增加尝试的次数，具体来说：
/data/hadoop/spark-1.6.0/bin/spark-shell –conf spark.port.maxRetries=100

参考：
http://blog.javachen.com/2015/06/07/spark-configuration.html?utm_source=tuicool&utm_medium=referral

报错信息

17/05/17 11:05:45 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: Service ‘SparkUI’ failed after 16 retries! Consider explicitly setting the appropriate port for the service ‘SparkUI’ (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils $.org$ apache $spark$ ui $JettyUtils$ $connect$ 1(JettyUtils.scala:252)
at org.apache.spark.ui.JettyUtils

a n o n f u n $ 5. a p p l y (J e t t y U t i l s . s c a l a : 262) a t o r g . a p a c h e . s p a r k . u i . J e t t y U t i l s

$anonfun$5.apply(JettyUtils.scala:262) at org.apache.spark.ui.JettyUtils$ anonfun

5.apply(JettyUtils.scala:262)atorg.apache.spark.util.Utils $5.apply(JettyUtils.scala:262) at org.apache.spark.util.Utils$

anonfun $anonfun$ startServiceOnPort

1.apply $1.apply$ mcVI

sp(Utils.scala:2062)atscala.collection.immutable.Range.foreach $sp(Utils.scala:2062) at scala.collection.immutable.Range.foreach$ mVc

sp(Range.scala:141)atorg.apache.spark.util.Utils $sp(Range.scala:141) at org.apache.spark.util.Utils$ .startServiceOnPort(Utils.scala:2053) at org.apache.spark.ui.JettyUtils

.startJettyServer(JettyUtils.scala:262)atorg.apache.spark.ui.WebUI.bind(WebUI.scala:136)atorg.apache.spark.SparkContext $.startJettyServer(JettyUtils.scala:262) at org.apache.spark.ui.WebUI.bind(WebUI.scala:136) at org.apache.spark.SparkContext$

anonfun $anonfun$ 13.apply(SparkContext.scala:481)
at org.apache.spark.SparkContext

a n o n f u n $ 13. a p p l y (S p a r k C o n t e x t . s c a l a : 481) a t s c a l a . O p t i o n . f o r e a c h (O p t i o n . s c a l a : 236) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t . < i n i t > (S p a r k C o n t e x t . s c a l a : 481) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p . c r e a t e S p a r k C o n t e x t (S p a r k I L o o p . s c a l a : 1017) a t $ l i n e 3. $ r e a d

$anonfun$13.apply(SparkContext.scala:481) at scala.Option.foreach(Option.scala:236) at org.apache.spark.SparkContext.<init>(SparkContext.scala:481) at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) at $line3.$read$ iwC

i w C . < i n i t > (< c o n s o l e > : 15) a t $ l i n e 3. $ r e a d

$iwC.<init>(<console>:15) at $line3.$read$ iwC.(:24)
at

line3. $line3.$ read.(:26)
at

line3. $line3.$ read

.(:30)at $.(:30) at$ line3.

read $read$ .()
at

line3. $line3.$ eval

.(:7)at $.(:7) at$ line3.

eval $eval$ .()
at

line3. $line3.$ eval.

print()atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:498)atorg.apache.spark.repl.SparkIMain $print() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.repl.SparkIMain$ ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain

Request.loadAndRun(SparkIMain.scala:1346)atorg.apache.spark.repl.SparkIMain.loadAndRunReq $Request.loadAndRun(SparkIMain.scala:1346) at org.apache.spark.repl.SparkIMain.loadAndRunReq$ 1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret

1(SparkILoop.scala:857)atorg.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)atorg.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)atorg.apache.spark.repl.SparkILoopInit $1(SparkILoop.scala:857) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) at org.apache.spark.repl.SparkILoopInit$

anonfun $anonfun$ initializeSpark

1.apply(SparkILoopInit.scala:125)atorg.apache.spark.repl.SparkILoopInit $1.apply(SparkILoopInit.scala:125) at org.apache.spark.repl.SparkILoopInit$

anonfun $anonfun$ initializeSpark

1.apply(SparkILoopInit.scala:124)atorg.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)atorg.apache.spark.repl.SparkILoopInit $1.apply(SparkILoopInit.scala:124) at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) at org.apache.spark.repl.SparkILoopInit$ class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop

a n o n f u n $ o r g $ a p a c h e $ s p a r k $ r e p l $ S p a r k I L o o p

$anonfun$org$apache$spark$repl$SparkILoop$ process

1 $1$

anonfun $anonfun$ apply

mcZ $mcZ$ sp

5.apply $5.apply$ mcV

sp(SparkILoop.scala:974)atorg.apache.spark.repl.SparkILoopInit $sp(SparkILoop.scala:974) at org.apache.spark.repl.SparkILoopInit$ class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit

class.postInitialization(SparkILoopInit.scala:108)atorg.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)atorg.apache.spark.repl.SparkILoop $class.postInitialization(SparkILoopInit.scala:108) at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) at org.apache.spark.repl.SparkILoop$

anonfun $anonfun$ org

apache $apache$ spark

repl $repl$ SparkILoop

p r o c e s s $ 1. a p p l y $ m c Z $ s p (S p a r k I L o o p . s c a l a : 991) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p

$process$1.apply$mcZ$sp(SparkILoop.scala:991) at org.apache.spark.repl.SparkILoop$ anonfun

org $org$ apache

spark $spark$ repl

SparkILoop $SparkILoop$

process $process$ 1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop

a n o n f u n $ o r g $ a p a c h e $ s p a r k $ r e p l $ S p a r k I L o o p

$anonfun$org$apache$spark$repl$SparkILoop$ process

1.apply(SparkILoop.scala:945)atscala.tools.nsc.util.ScalaClassLoader $1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader$ .savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org

apache $apache$ spark

repl $repl$ SparkILoop

p r o c e s s (S p a r k I L o o p . s c a l a : 945) a t o r g . a p a c h e . s p a r k . r e p l . S p a r k I L o o p . p r o c e s s (S p a r k I L o o p . s c a l a : 1059) a t o r g . a p a c h e . s p a r k . r e p l . M a i n $ . m a i n (M a i n . s c a l a : 31) a t o r g . a p a c h e . s p a r k . r e p l . M a i n . m a i n (M a i n . s c a l a) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e M e t h o d A c c e s s o r I m p l . i n v o k e (N a t i v e M e t h o d A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g M e t h o d A c c e s s o r I m p l . i n v o k e (D e l e g a t i n g M e t h o d A c c e s s o r I m p l . j a v a : 43) a t j a v a . l a n g . r e f l e c t . M e t h o d . i n v o k e (M e t h o d . j a v a : 498) a t o r g . a p a c h e . s p a r k . d e p l o y . S p a r k S u b m i t $ . o r g $ a p a c h e $ s p a r k $ d e p l o y $ S p a r k S u b m i t

$process(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$ runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit

.doRunMain $.doRunMain$ 1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit

.submit(SparkSubmit.scala:206)atorg.apache.spark.deploy.SparkSubmit $.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$ .main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)