下载
文档地址[url]http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html[/url]
[url]http://www.diaryfolio.com/hadoop-install-steps/[/url]
[size=large][b]前期准备[/b][/size]
解压到指定目录
添加hadoop用户和用户组
查看hadoop用户
将解压的hadoop文件目录所属用户改为hadoop
将hadoop添加到sudo中
在末尾添加一行
保存
切换到hadoop用户
[size=large][b]环境变量和SSH配置[/b][/size]
以下操作都使用hadoop用户
修改环境变量
保存
使环境变量立即生效
设置SSH
[size=large][b]
hadoop配置文件[/b][/size]
SSH端口不是默认的22端口,修改环境配置文件
追加一行
SSH认证文件权限不正确
切换到su hadoop 到/home/hadoop目录
执行
创建 hadoop-2.3.0/tmp目录
修改hadoop-2.3.0/etc/hadoop/mapred-site.xml
修改四个配置文件
[size=large][b]hadoop启动和停止[/b][/size]
格式化HDFS文件系统
启动系统
查看运行日志:
[quote] less logs/hadoop-hadoop-datanode-UAT.log[/quote]
停止系统
查看Java虚拟机里运行的hadoop进程
[size=large][b]运行wordcount实例[/b][/size]
创建空文件
复制以下内容进去
启动hadoop,
执行wordcount命令
查看生成的文件
查看统计结果
删除已生成的结果文件
运行状态:
webl界面
[url]http://192.168.1.22:50075/dataNodeHome.jsp[/url]
hfs文件管理
[url]http://192.168.1.22:50070/dfshealth.html[/url]
[url]http://192.168.1.22:50090/status.jsp[/url]
节点管理
[url]http://192.168.1.22:8042/node[/url]
应用管理
[url]http://192.168.1.22/node:8088[/url]
hadoop缺省端口
http://hsrong.iteye.com/blog/1374734
[url="http://powertech.iteye.com/admin/blogs/2082204"]hadoop2.3集群[/url]
文档地址[url]http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html[/url]
[url]http://www.diaryfolio.com/hadoop-install-steps/[/url]
[size=large][b]前期准备[/b][/size]
解压到指定目录
tar -zxvf hadoop-2.3.0.tar.gz -C /data/javadev
添加hadoop用户和用户组
useradd hadoop
usermod -g hadoop hadoop
查看hadoop用户
id hadoop
将解压的hadoop文件目录所属用户改为hadoop
chown -R hadoop:hadoop hadoop-2.3.0/
将hadoop添加到sudo中
visudo
在末尾添加一行
hadoop ALL=(ALL) ALL
保存
切换到hadoop用户
su - hadoop
[size=large][b]环境变量和SSH配置[/b][/size]
以下操作都使用hadoop用户
修改环境变量
vi .bashrc
export JAVA_HOME=/usr/java/jdk1.6.0_31/
#hadoop var
HADOOP_COMMON_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_HDFS_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_MAPRED_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_YARN_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_CONF_DIR=/data01/javaapp/hadoop-2.3.0/etc/hadoop
YARN_CONF_DIR=/data01/javaapp/hadoop-2.3.0/etc/hadoop
export HADOOP_COMMON_HOME HADOOP_HDFS_HOME HADOOP_MAPRED_HOME HADOOP_YARN_HOME HADOOP_CONF_DIR YARN_CONF_DIR
export PATH=$PATH:$HADOOP_COMMON_HOME/bin
保存
使环境变量立即生效
source .bashrc
设置SSH
ssh-keygen -t rsa -P ""
cat id_dsa.pub >> ~/.ssh/authorized_keys
chmod 644 authorized_keys
cd ~
chmod 700 .ssh
[size=large][b]
hadoop配置文件[/b][/size]
connect to host localhost port 22: Connection refused
SSH端口不是默认的22端口,修改环境配置文件
hadoop-2.3.0/etc/hadoop/hadoop-env.sh
追加一行
export HADOOP_SSH_OPTS="-p <num>"
The authenticity of host 'localhost (127.0.0.1)' can't be established.
SSH认证文件权限不正确
切换到su hadoop 到/home/hadoop目录
执行
chmod 644 authorized_keys
cd ~
chmod 700 .ssh
创建 hadoop-2.3.0/tmp目录
mkdir tmp
修改hadoop-2.3.0/etc/hadoop/mapred-site.xml
修改四个配置文件
#hadoop-2.3.0/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/data01/javaapp/hadoop-2.3.0/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>
#vi etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
#vi etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
#vi etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
[size=large][b]hadoop启动和停止[/b][/size]
格式化HDFS文件系统
$HADOOP_HOME/bin/hadoop namenode -format
启动系统
$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh start secondarynamenode
$HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
$HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
查看运行日志:
[quote] less logs/hadoop-hadoop-datanode-UAT.log[/quote]
停止系统
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver
$HADOOP_HOME/sbin/yarn-daemon.sh stop nodemanager
$HADOOP_HOME/sbin/yarn-daemon.sh stop resourcemanager
$HADOOP_HOME/sbin/hadoop-daemon.sh stop secondarynamenode
$HADOOP_HOME/sbin/hadoop-daemon.sh stop datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh stop namenode
查看Java虚拟机里运行的hadoop进程
# Java Virtual Machine Process Status Tool (jps) acts like a ps command and
# shows various java process running. Hence running would show all hadoop processes.
hduser@diaryfoliovm:/opt/hadoop$ jps
2243 TaskTracker
2314 JobTracker
1923 DataNode
2895 SecondaryNameNode
1234 Jps
1788 NameNode
[size=large][b]运行wordcount实例[/b][/size]
创建空文件
vi hadoop-2.3.0/tmp/input
复制以下内容进去
Read: Father of Santa Barbara Victim Sobs and Rails Against Son's Death
That "last chance" turned bleak – a night that reflected his ambitions,
fury and warped perspectives. It became a flashpoint leading up to last
Friday's attacks that left six others dead and 13 injured.
WATCH: Massive Crowd Comes to Mourn Santa Barbara Shooting Victims
Rodger bought a bottle of vodka that night, taking a few shots for
courage, maybe downing one too many. Other students were partying –
"good looking popular kids," as he identified them. Without the buzz,
he would have been too intimidated to mingle.
启动hadoop,
$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh start secondarynamenode
$HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
$HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver
执行wordcount命令
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount file:///data01/javaapp/hadoop-2.3.0/tmp/input output2
查看生成的文件
bin/hadoop fs -ls -R output2
14/05/28 14:06:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r-- 1 hadoop supergroup 0 2014-05-28 13:58 output2/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 4574 2014-05-28 13:58 output2/part-r-00000
查看统计结果
bin/hadoop fs -cat output2/part-r-00000
删除已生成的结果文件
bin/hadoop fs -rmr output*
运行状态:
bin/hadoop dfsadmin -report
webl界面
[url]http://192.168.1.22:50075/dataNodeHome.jsp[/url]
hfs文件管理
[url]http://192.168.1.22:50070/dfshealth.html[/url]
[url]http://192.168.1.22:50090/status.jsp[/url]
节点管理
[url]http://192.168.1.22:8042/node[/url]
应用管理
[url]http://192.168.1.22/node:8088[/url]
hadoop缺省端口
http://hsrong.iteye.com/blog/1374734
[url="http://powertech.iteye.com/admin/blogs/2082204"]hadoop2.3集群[/url]