hue与Hadoop的集成

hue与Hadoop的集成

1、修改Hadoop相关配置文件

hdfs-site.xml

dfs.webhdfs.enabled -》默认是开启，所以不再配置

<property>

<name>dfs.permissions.enabled</name>

<value>false</value>

</property>

core-site.xml

<property> 配置hue的访问hdfs的权限 （oozie）

<name>hadoop.proxyuser.hue.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hue.groups</name>

<value>*</value>

</property>

2、修改hue的主配置文件里的hdfs、yarn模块

[[hdfs_clusters]]

# HA support by using HttpFs

[[[default]]]

# Enter the filesystem uri

fs_defaultfs=hdfs://blue01.mydomain:8020

# NameNode logical name.

## logical_name=

# Use WebHdfs/HttpFs as the communication mechanism.

# Domain should be the NameNode or HttpFs host.

# Default port is 14000 for HttpFs.

webhdfs_url=http://blue01.mydomain:50070/webhdfs/v1

hadoop_hdfs_home=/opt/modules/hadoop-2.5.0-cdh5.3.6

hadoop_bin=/opt/modules/hadoop-2.5.0-cdh5.3.6/bin

hadoop_conf_dir=/opt/modules/hadoop-2.5.0-cdh5.3.6/etc/hadoop

[[yarn_clusters]]

[[[default]]]

# Enter the host on which you are running the ResourceManager

resourcemanager_host=blue01.mydomain

# The port where the ResourceManager IPC listens on

resourcemanager_port=8032

# Whether to submit jobs to this cluster

submit_to=True

# Resource Manager logical name (required for HA)

## logical_name=

# Change this if your YARN cluster is Kerberos-secured

## security_enabled=false

# URL of the ResourceManager API

resourcemanager_api_url=http://blue01.mydomain:8088

# URL of the ProxyServer API

proxy_api_url=http://blue01.mydomain:8088

# URL of the HistoryServer API

history_server_api_url=http://blue01.mydomain:19888

3、重启Hadoop服务集成及hue server

$ sbin/stop-all.sh

$ sbin/start-all.sh

$ build/env/bin/supervisor

右上角 管理hdfs ：

增删改查、上传等

右上角 管理作业 ：

$ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0-cdh5.3.6.jar wordcount /user/wc.txt /user/output1

注意:需要开启hdfs yarn historyserver服务

$ sbin/stop-all.sh

$ sbin/start-all.sh

$ sbin/mr-jobhistory-daemon.sh start historyserver