hawq 集成yarn

安装好hawq，参考：http://blog.youkuaiyun.com/u011563666/article/details/78814520

hawq配置yarn支持分为yarn和yarn HA。本文简单介绍hawq配置yarn。配置yarn HA请参考：

http://hawq.incubator.apache.org/docs/userguide/2.1.0.0-incubating/resourcemgmt/YARNIntegration.html#highlyavailableyarn

1. 安装好hadoop HDFS、YARN系统

2. 修改yarn配置文件yarn-site.xml

     添加或修改：

   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
      <name>yarn.resourcemanager.scheduler.class</name>
      <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>
    <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>xxx.xxx.xxx.xxx/resouremanager hostname</value>
    </property>

3. 修改capacity-scheduler.xml

      给调度器新增一个队列：hawqque

       

<property>
      <name>yarn.scheduler.capacity.root.queues</name>
      <value>default,hawqque</value>
   </property>

     为hawqque配置参数

     

 <property>
    <name>yarn.scheduler.capacity.root.hawqque.capacity</name>
    <value>50</value>
    <description>hawqque queue target capacity.</description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.hawqque.user-limit-factor</name>
    <value>1</value>
    <description>
      hawqque queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.hawqque.maximum-capacity</name>
    <value>80</value>
    <description>
      The maximum capacity of the hawqque queue.
    </description>
  </property>
 <property>
    <name>yarn.scheduler.capacity.root.hawqque.state</name>
    <value>RUNNING</value>
    <description>
      The state of the hawqque queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

      配置队列权限：

    

 <property>
    <name>yarn.scheduler.capacity.root.hawqque.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the hawqque queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.hawqque.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the hawqque queue.
    </description>
  </property>

     4.配置hawq，启用yarn

      修改/usr/local/apache-hawq/etc/hawq-size.xml

     

vim /usr/local/apache-hawq/etc/hawq-site.xml

       修改hawq_global_rm_type参数为yarn，同时配置好yarn参数地址，及hawq的调度对了名称hawqque

       

<property>
		<name>hawq_global_rm_type</name>
		<value>yarn</value>
		<description>The resource manager type to start for allocating resource.
								 'none' means hawq resource manager exclusively uses whole
								 cluster; 'yarn' means hawq resource manager contacts YARN
								 resource manager to negotiate resource.
		</description>
</property>   

     

        <property>
                <name>hawq_rm_yarn_address</name>
                <value>x.x.x.x/resouremanager hostname:8032</value>
                <description>The address of YARN resource manager server.</description>
        </property>


        <property>
                <name>hawq_rm_yarn_scheduler_address</name>
                <value>x.x.x.x/resourcemanager hostname:8030</value>
                <description>The address of YARN scheduler server.</description>
        </property>


        <property>
                <name>hawq_rm_yarn_queue_name</name>
                <value>hawqque</value>
                <description>The YARN queue name to register hawq resource manager.</description>
        </property>

      5. 初始化hawq集群

           保证hdfs、yarn系统运行正常，执行：

         

hawq init cluster

    6. 查看yarn页面，确认启动成功

       http://xxx.xx.xxx.xx:8088/

      查看scheduler，查看Applications->running，会有hawq的YARN类型Job。