Spark在Yarn上的动态资源分配

参考地址：http://spark.apache.org/docs/1.5.2/job-scheduling.html#configuration-and-setup

1.配置hadoop/etc/yarn-site.xml

1. <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle,spark_shuffle</value>
   </property>
2. 
   <property>
     <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
     <value>org.apache.spark.network.yarn.YarnShuffleService</value>
   </property>
3. 
   
4. <!-- 默认端口号 -->
5. <property>

  <name>spark.shuffle.service.port</name>
  <value>7337</value>
</property>

2.拷贝spark-xxx-yarn-shuffle.jar到指定目录

拷贝spark-xxx-yarn-shuffle.jar到 hadoop/share/hadoop/yarn/里面

3.配置spark-default.conf

spark.shuffle.service.enabled  true

spark.shuffle.service.prot         7337

Dynamic Allocation

Property Name