解决Spark on YARN时大量jar乱飞的情况

目录

一、问题重现

二、解决方法

三、结果验证

---

一、问题重现

使用Spark on YARN运行自带的SparkPi作业

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode client \
/opt/app/spark-2.4.5-bin-2.6.0-cdh5.16.2/examples/jars/spark-examples_2.12-2.4.5.jar \
2

输出日志

WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
INFO yarn.Client: Uploading resource file:/tmp/spark-a7f4a566-9d21-43cf-8388-41698316838e/__spark_libs__855332709144911684.zip -> hdfs://mycluster/user/root/.sparkStaging/application_1588347637771_0011/__spark_libs__855332709144911684.zip
INFO yarn.Client: Uploading resource file:/tmp/spark-a7f4a566-9d21-43cf-8388-41698316838e/__spark_conf__9173892461601825482.zip -> hdfs://mycluster/user/root/.sparkStaging/application_1588347637771_0011/__spark_conf__.zip

提示我们既没有设置spark.yarn.archive也没有设置spark.yarn.jars，所以spark