Hadoop系列之-安装与Wordcount例子

一、确保主机名配置文件如下

vi /etc/hosts
vi /etc/sysconfig/network

[root@steven ~]# rpm -qa|grep java
java-1.6.0-openjdk-1.6.0.35-1.13.7.1.el6_6.x86_64
java-1.7.0-openjdk-1.7.0.79-2.5.5.4.el6.x86_64
tzdata-java-2015e-1.el6.noarch
[root@steven ~]# rpm -e --nodeps tzdata-java-2015e-1.el6.noarch
[root@steven ~]# rpm -e --nodeps java-1.6.0-openjdk-1.6.0.35-1.13.7.1.el6_6.x86_64
[root@steven ~]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.79-2.5.5.4.el6.x86_64
[root@steven ~]# java -version
-bash: java: command not found

二、创建Hadoop用户以及开启SSH

useradd hadoop
passwd hadoop

su - hadoop
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
exit

遇到要输入yes或者no就输入yes,其他回车就OK

三、配置文件修改

<strong>core-site.xml</strong>

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

<strong>hdfs-site.xml (目录要创建好)</strong>

<configuration>
<property>
 <name>dfs.replication</name>
 <value>1</value>
</property>

<property>
  <name>dfs.name.dir</name>
    <value>file:///opt/hadoop-2.6.3/hadoopdata/hdfs/namenode</value>
</property>

<property>
  <name>dfs.data.dir</name>
    <value>file:///opt/hadoop-2.6.3/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

<strong>mapred-site.xml</strong>

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>
</configuration>

<strong>yarn-site.xml</strong>

<configuration>
 <property>
  <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
 </property>
</configuration>

四、配置.bashrc

export JAVA_HOME=/opt/jdk1.8.0_101
export CLASSPATH=.:%JAVA_HOME%/lib/dt.jar:%JAVA_HOME%/lib/tools.jar
export HADOOP_HOME=/opt/hadoop-2.6.3
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
======================================================================
source ~/.bashrc //配置文件生效

五、格式化以及开启服务

hdfs namenode -format

start-all.sh

接着就是创建/home/hadoop(创建的用户名)/input文件夹，然后再里面创建f1.txt文件，里面输入英语单词
在运行的hadoop中创建文件夹
hadoop dfs -mkdir -p /home/hadoop/input
查看是否成功创建文件夹
hadoop dfs -ls /home/hadoop
如果我们能看到类似drwr-xr-x ....../home/hadoop/input字样那表明我们这一步已经成功了

把文件复制到hadoop中(在hadoop用户名目录下)
hadoop dfs -put input/* /home/hadoop/input
查看文件
hadoop dfs -ls /home/hadoop/input
hadoop dfs -cat /home/hadoop/input/f1.txt

http://192.168.3.59:50070 然后点击工具，点击浏览文件系统，可以看到/home/hadoop/input目录

六、运行自带的例子Wordcount

在hadoop安装目录下执行

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar wordcount /home/hadoop/input /home/hadoop/output

发现output文件夹也生成了

输入hadoop dfs -cat /home/hadoop/output/part-r-00000 查看单词统计结果 

hadoop-env.sh(etc/hadoop/hadoop-env.sh)

添加：

export PATH=${JAVA_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

重新测试

hadoop dfs -rm -r /user/root/output

hadoop dfs -rm -r /user/root/input/*

查看文件大小

du -sh *

ls -ll

hadoop-streaming-2.7.2.jar位置

/opt/hadoop-2.7.2/share/hadoop/tools/lib/hadoop-streaming-2.7.2.jar