spark的常用参数配置（local、yarn）

spark配置的优先级

虽然，spark配置的优先级是这样的：
代码中SparkConf的设置(最高的优先级) > spark-submit --选项 > spark-defaults.conf配置 > spark-env.sh配置 > 默认值
但实际测试下来发现，spark-submit的优先级会比在脚本中配置SparkConf要高，因此，在实际的编程过程中，推荐优先使用spark-submit的方式。

local模式

/opt/spark2/bin/spark-submit \
--master local \
--driver-memory 8g \
--executor-memory 16g \
--num-executors 4 \
--executor-cores 4 \
--conf spark.default.parallelism=100 \
spark_test.py

yarn模式

client方式

/opt/spark2/bin/spark-submit \
--master yarn \
--deploy-mode client \
--driver-memory 8g \
--executor-memory 16g \
--num-executors 4 \
--executor-cores 4 \
--conf spark.default.parallelism=100 \
--conf spark.executorEnv.PYTHONPATH=pyspark.zip:py4j-0.10.7-src.zip \
--conf spark.yarn.dist.files=file:/opt/spark2/python/lib/pyspark.zip,file:/opt/spark2/python/lib/py4j-0.10.7-src.zip \
--conf spark.hadoop.hive.exec.dynamic.partition=true \
--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict \
spark_test.py

cluster方式

/opt/spark2/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 8g \
--executor-memory 16g \
--num-executors 4 \
--executor-cores 4 \
--conf spark.default.parallelism=100 \
--conf spark.executorEnv.PYTHONPATH=pyspark.zip:py4j-0.10.7-src.zip \
--conf spark.yarn.dist.files=file:/opt/spark2/python/lib/pyspark.zip,file:/opt/spark2/python/lib/py4j-0.10.7-src.zip \
--conf spark.hadoop.hive.exec.dynamic.partition=true \
--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict \
--py-files='/export/uddev/udhuangj/self_study/python_cluster.zip' \
spark_test.py