提交spark任务偶尔报错 org.apache.spark.SparkException: A master URL must be set in your configuration

最新推荐文章于 2025-06-27 21:32:28 发布

原创

最新推荐文章于 2025-06-27 21:32:28 发布 · 606 阅读

2 ·

CC 4.0 BY-SA版权

在提交Spark任务时遇到报错，错误信息为'org.apache.spark.SparkException: A master URL must be set in your configuration'。问题可能出在pom.xml文件中依赖库缺少<scope>compile</scope>，以及非main函数初始化问题。通过查看链接：https://www.cnblogs.com/barneywill/p/10109122.html 找到了解决方案。

问题一：pox.xml里外部以来库没有加入<scope>compile</scope>导致无法初始化

问题二：main函数以外的方法有时也无法初始化

错误信息：

20/11/24 22:32:00 INFO DAGScheduler: ResultStage 0 (take at Base64UserEmb.scala:57) failed in 16.761 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 9.10.8.90, executor 3): java.lang.ExceptionInInitializerError
	at com.tencent.ieg.dm.pltv.feature.Base64UserEmb$$anonfun$3.apply(Base64UserEmb.scala:52)
	at com.tencent.ieg.dm.pltv.feature.Base64UserEmb$$anonfun$3.apply(Base64UserEmb.scala:48)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:394)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.Array

最低0.47元/天解锁文章