spark在yarn上运行遇见的错误

最新推荐文章于 2023-07-22 15:36:05 发布
原创
最新推荐文章于 2023-07-22 15:36:05 发布 · 1.8k 阅读
2 ·
CC 4.0 BY-SA版权
Spark代码运行出错

19/06/19 22:29:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/06/19 22:29:30 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/06/19 22:29:31 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
19/06/19 22:29:31 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
19/06/19 22:29:31 INFO yarn.Client: Setting up container launch context for our AM
19/06/19 22:29:31 INFO yarn.Client: Setting up the launch environment for our AM container
19/06/19 22:29:31 INFO yarn.Client: Preparing resources for our AM container
19/06/19 22:29:34 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
19/06/19 22:29:44 INFO yarn.Client: Uploading resource file:/tmp/spark-4ebf7d5b-61e2-4768-95d4-dfb97d199e56/__spark_libs__5213362222344047075.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/__spark_libs__5213362222344047075.zip
19/06/19 22:30:01 INFO yarn.Client: Uploading resource file:/home/hadoop01/count/spark-study-core-1.0-SNAPSHOT-jar-with-dependencies.jar -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/spark-study-core-1.0-SNAPSHOT-jar-with-dependencies.jar
19/06/19 22:30:12 INFO yarn.Client: Uploading resource file:/tmp/spark-4ebf7d5b-61e2-4768-95d4-dfb97d199e56/__spark_conf__1038658387679626348.zip -> hdfs://bd1901/user/hadoop01/.sparkStaging/application_1560952784300_0002/__spark_conf__.zip
19/06/19 22:30:12 INFO spark.SecurityManager: Changing view acls to: hadoop01
19/06/19 22:30:12 INFO spark.SecurityManager: Changing modify acls to: hadoop01
19/06/19 22:30:12 INFO spark.SecurityManager: Changing view acls groups to: 
19/06/19 22:30:12 INFO spark.SecurityManager: Changing modify acls groups to: 
19/06/19 22:30:12 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop01); groups with view permissions: Set(); users  with modify permissions: Set(hadoop01); groups with modify permissions: Set()
19/06/19 22:30:12 INFO yarn.Client: Submitting application application_1560952784300_0002 to ResourceManager
19/06/19 22:30:12 INFO impl.YarnClientImpl: Submitted application application_1560952784300_0002
19/06/19 22:30:13 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:14 INFO yarn.Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1560954612808
         final status: UNDEFINED
         tracking URL: http://hadoop02:8088/proxy/application_1560952784300_0002/
         user: hadoop01
19/06/19 22:30:15 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:16 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:17 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:30:18 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
。。。。。。
19/06/19 22:31:34 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:31:35 INFO yarn.Client: Application report for application_1560952784300_0002 (state: ACCEPTED)
19/06/19 22:31:36 INFO yarn.Client: Application report for application_1560952784300_0002 (state: FAILED)
19/06/19 22:31:36 INFO yarn.Client:
client token: N/A
         diagnostics: Application application_1560952784300_0002 failed 2 times due to AM Container for appattempt_1560952784300_0002_000002 exited with  exitCode: 15
For more detailed output, check application tracking page:http://hadoop02:8088/cluster/app/application_1560952784300_0002Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e54_1560952784300_0002_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
        at org.apache.hadoo