Spark on K8s: Failed to connect to spark-master:7077

原创

已于 2024-12-28 13:10:11 修改 · 994 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据 #k8s

于 2024-12-28 13:00:05 首次发布

异常信息如下：

spark 08:50:17.93
spark 08:50:17.94 Welcome to the Bitnami spark container
spark 08:50:17.94 Subscribe to project updates by watching https://github.com/bitnami/containers
spark 08:50:17.95 Submit issues and feature requests at https://github.com/bitnami/containers/issues
spark 08:50:17.95

24/12/27 08:50:21 INFO SparkContext: Running Spark version 3.3.1
24/12/27 08:50:21 INFO ResourceUtils: ==============================================================
24/12/27 08:50:21 INFO ResourceUtils: No custom resources configured for spark.driver.
24/12/27 08:50:21 INFO ResourceUtils: ==============================================================
24/12/27 08:50:21 INFO SparkContext: Submitted application: Spark Pi
24/12/27 08:50:22 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
24/12/27 08:50:22 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
24/12/27 08:50:22 INFO ResourceProfileManager: Added ResourceProfile id: 0
24/12/27 08:50:22 INFO SecurityManager: Changing view acls to: spark
24/12/27 08:50:22 INFO SecurityManager: Changing modify acls to: spark
24/12/27 08:50:22 INFO SecurityManager: Changing view acls groups to:
24/12/27 08:50:22 INFO SecurityManager: Changing modify acls groups to:
24/12/27 08:50:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
24/12/27 08:50:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/12/27 08:50:23 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
24/12/27 08:50:23 INFO SparkEnv: Registering MapOutputTracker
24/12/27 08:50:23 INFO SparkEnv: Registering BlockManagerMaster
24/12/27 08:50:23 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
24/12/27 08:50:23 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
24/12/27 08:50:23 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
24/12/27 08:50:23 INFO DiskBlockManager: Created local directory at /var/data/spark-3016eb8d-b215-46f9-b66a-fa05f3143941/blockmgr-5ad552e1-3ce2-4b39-b060-1a4cfb9df197
24/12/27 08:50:23 INFO MemoryStore: MemoryStore started with capacity 413.9 MiB
24/12/27 08:50:23 INFO SparkEnv: Registering OutputCommitCoordinator
24/12/27 08:50:23 INFO Utils: Successfully started service 'SparkUI' on port 4040.
24/12/27 08:50:23 INFO SparkContext: Added JAR local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.3.1.jar at file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.3.1.jar with timestamp 1735289421910
24/12/27 08:50:23 INFO SparkContext: The JAR local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.3.1.jar at file:/opt/bitnami/spark/examples/jars/spark-examples_2.12-3.3.1.jar has been added already. Overwriting of added jar is not supported in the current version.
24/12/27 08:50:24 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master:7077...
24/12/27 08:50:24 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master spark-master:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEn