hadoop使用指南

hadoop安装

官网：http://hadoop.apache.org/
下载并安装到opt目录下（3.x要求jdk1.8版本）
tar -zxvf hadoop-3.2.0.tar.gz

常用命令
jps

格式化
bin/hdfs namenode -format

启动namenode
sbin/hadoop-daemon.sh start namenode

启动datanode
sbin/hadoop-daemon.sh start datanode

启动nodemanager
sbin/yarn-daemon.sh start nodemanager

启动resourcemanager
sbin/yarn-daemon.sh start resourcemanager

启动historyserver
sbin/mr-jobhistory-daemon.sh start historyserver

dfs命令
bin/hdfs dfs 后面加linux命令

删除output
bin/hdfs dfs -rm -r /user/output

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /user/input /user/output

本地模式

• grep测试
  进入hadoop目录
  cd hadoop

创建input文件夹
mkdir input

将etc里的所有xml文件复制到input文件夹里（此处etc是hadoop目录下的etc）
cp etc/hadoop/*.xml input/

执行以下命令（测试 获取符合’dfs[a-z.]+'该正则表达式的个数（可在output文件夹里一个文件看到））
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar grep input/ output ‘dfs[a-z.]+’

注释：使用该命令时output文件夹不能有，他会自行创建，否则会报错

• WordCount测试
  进入hadoop目录
  cd hadoop

创建wcinput文件夹
mkdir wcinput
cd wc.input

创建wc.input文件并编辑
touch wc.input
vim wc.input

添加以下内容
hadoop yarn
hadoop mapreduce
hzdy
hzdy

测试
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount wcinput wcoutput
查看
cat wcoutput/part-r-00000

伪分布式HDFS

配置etc/hadoop/core-site.xml
vim etc/hadoop/core-site.xml
如下

    <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop100:9000</value>
    </property>

    <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop-3.2.0/data/tmp</value>
    </property>

配置etc/hadoop/hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
如下

配置etc/hadoop/hadoop-env.sh（不配置的集群不能识别）
export JAVA_HOME=/opt/jdk1.8.0_201

启动集群

（a）格式化NameNode(第一次启动时格式化，以后就不要总格式化了)

bin/hdfs namenode  -format

（b）启动NameNode

sbin/hadoop-daemon.sh start namenode

（c）启动DataNode

sbin/hadoop-daemon.sh start datanode

查看集群

jps

访问localhost:9870（2.x版本端口号为50070，3.x版本端口号为9870）
点击Utilities下的Browse the file system
里面相当于一个linux文件目录

创建目录

bin/hdfs dfs -mkdir -p /user/input

浏览器可以看到新建的文件目录

上传文件（前面参数是上传文件路径 后面参数是目的路径）

bin/hdfs dfs -put wcinput/wc.input /user/input

浏览器可以看到上传的文件

值得注意的是hdfs的根目录是hdfs://localhost:9000/user/root，创建的文件都在这下面
这时候可以测试一下：

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar  wordcount /user/input /user/output

YARN启动mapreduce

编辑 yarn-env配置java_home

vim etc/hadoop/yarn-env.sh

配置etc/hadoop/yarn-site.xml

vim etc/hadoop/yarn-site.xml

配置etc/hadoop/mapred-env.sh

vim etc/hadoop/mapred-env.sh

配置etc/hadoop/mapred-site.xml(使其在yarn上运行，默认是local)

vim  etc/hadoop/mapred-site.xml

内容如下（HADOOP_MAPRED_HOME的值可以通过hadoop classpath获得再贴上去）

        <configuration>
            <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
            </property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
            </property>

    <property>
    <name>yarn.app.mapreduce.am.env</name>
    <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
    </property>

    <property>
    <name>mapreduce.map.env</name>
    <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
    </property>

    <property>
    <name>mapreduce.reduce.env</name>
    <value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
    </property>

<property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop100:10020</value>
    </property>

	<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop100:19888</value>
    </property>
</configuration>

启动集群

启动之前先确认nameNode和dataNode已经运行

开启resourcemanager

sbin/yarn-daemon.sh start resourcemanager

开启nodemanager

sbin/yarn-daemon.sh start nodemanager

开启历史服务器historyserver

sbin/mr-jobhistory-daemon.sh  start historyserver

浏览器打开http://localhost:8088查看程序运行进程

删除之前的output

bin/hdfs dfs -rm -r  /user/output

测试：

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar  wordcount /user/input /user/output

hadoop（完全分布式配置)

进入hadoop主目录下的etc/hadoop/修改配置文件

配置core-site.xml地址为主ip

配置hadoop-env.sh的java路径

配置hdfs-site.xml辅助名称节点主机（辅助节点是另外一台主机）

配置yarn-env.sh的java路径

配置yarn-site.xml的resourceManager地址
地址为除主机和辅助节点外的另外一台

配置mapred-env.sh的java路径

配置mapred-site.xml

分发所有hadoop配置文件
xsync /opt/hadoop-3.2.0/
分发完成后其他主机不需要修改内容了

启动集群

清空三台主机的logs
rm -rf data/logs/

格式化namenode（只格式化master）
bin/hdfs namenode -format

开启namenode（只开启master）
sbin/hadoop-daemon.sh start namenode

开启datanode(三台都开)
sbin/hadoop-daemon.sh start datanode