hadoop 2.3.0 配置 Single Node Cluster

下载

文档地址[url]http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html[/url]

[url]http://www.diaryfolio.com/hadoop-install-steps/[/url]


[size=large][b]前期准备[/b][/size]

解压到指定目录
tar -zxvf hadoop-2.3.0.tar.gz -C /data/javadev


添加hadoop用户和用户组
useradd hadoop
usermod -g hadoop hadoop


查看hadoop用户
id hadoop


将解压的hadoop文件目录所属用户改为hadoop
chown -R hadoop:hadoop hadoop-2.3.0/


将hadoop添加到sudo中
visudo

在末尾添加一行
hadoop          ALL=(ALL)               ALL

保存

切换到hadoop用户
su - hadoop



[size=large][b]环境变量和SSH配置[/b][/size]

以下操作都使用hadoop用户

修改环境变量

vi .bashrc



export JAVA_HOME=/usr/java/jdk1.6.0_31/

#hadoop var
HADOOP_COMMON_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_HDFS_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_MAPRED_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_YARN_HOME=/data01/javaapp/hadoop-2.3.0
HADOOP_CONF_DIR=/data01/javaapp/hadoop-2.3.0/etc/hadoop
YARN_CONF_DIR=/data01/javaapp/hadoop-2.3.0/etc/hadoop

export HADOOP_COMMON_HOME HADOOP_HDFS_HOME HADOOP_MAPRED_HOME HADOOP_YARN_HOME HADOOP_CONF_DIR YARN_CONF_DIR

export PATH=$PATH:$HADOOP_COMMON_HOME/bin


保存

使环境变量立即生效

source .bashrc


设置SSH
ssh-keygen -t rsa -P ""   

cat id_dsa.pub >> ~/.ssh/authorized_keys


chmod 644 authorized_keys
cd ~
chmod 700 .ssh


[size=large][b]
hadoop配置文件[/b][/size]

connect to host localhost port 22: Connection refused

SSH端口不是默认的22端口,修改环境配置文件
 hadoop-2.3.0/etc/hadoop/hadoop-env.sh

追加一行
export HADOOP_SSH_OPTS="-p <num>"



The authenticity of host 'localhost (127.0.0.1)' can't be established.

SSH认证文件权限不正确
切换到su hadoop 到/home/hadoop目录
执行

chmod 644 authorized_keys
cd ~
chmod 700 .ssh



创建 hadoop-2.3.0/tmp目录
mkdir tmp


修改hadoop-2.3.0/etc/hadoop/mapred-site.xml


修改四个配置文件

#hadoop-2.3.0/etc/hadoop/core-site.xml

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/data01/javaapp/hadoop-2.3.0/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
</property>
</configuration>




#vi etc/hadoop/mapred-site.xml

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
</property>
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>



#vi etc/hadoop/hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>


#vi etc/hadoop/yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>



[size=large][b]hadoop启动和停止[/b][/size]

格式化HDFS文件系统

$HADOOP_HOME/bin/hadoop namenode -format  


启动系统
$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh start secondarynamenode

$HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
$HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver


查看运行日志:
[quote] less logs/hadoop-hadoop-datanode-UAT.log[/quote]


停止系统
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh stop historyserver
$HADOOP_HOME/sbin/yarn-daemon.sh stop nodemanager
$HADOOP_HOME/sbin/yarn-daemon.sh stop resourcemanager

$HADOOP_HOME/sbin/hadoop-daemon.sh stop secondarynamenode
$HADOOP_HOME/sbin/hadoop-daemon.sh stop datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh stop namenode



查看Java虚拟机里运行的hadoop进程
# Java Virtual Machine Process Status Tool (jps) acts like a ps command and 
# shows various java process running. Hence running would show all hadoop processes.
hduser@diaryfoliovm:/opt/hadoop$ jps
2243 TaskTracker
2314 JobTracker
1923 DataNode
2895 SecondaryNameNode
1234 Jps
1788 NameNode



[size=large][b]运行wordcount实例[/b][/size]

创建空文件
vi hadoop-2.3.0/tmp/input 

复制以下内容进去
Read: Father of Santa Barbara Victim Sobs and Rails Against Son's Death
That "last chance" turned bleak – a night that reflected his ambitions,
fury and warped perspectives. It became a flashpoint leading up to last
Friday's attacks that left six others dead and 13 injured.
WATCH: Massive Crowd Comes to Mourn Santa Barbara Shooting Victims
Rodger bought a bottle of vodka that night, taking a few shots for
courage, maybe downing one too many. Other students were partying –
"good looking popular kids," as he identified them. Without the buzz,
he would have been too intimidated to mingle.



启动hadoop,

$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh start secondarynamenode

$HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager
$HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver


执行wordcount命令
 bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount file:///data01/javaapp/hadoop-2.3.0/tmp/input output2


查看生成的文件
 bin/hadoop fs  -ls -R output2

 14/05/28 14:06:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r-- 1 hadoop supergroup 0 2014-05-28 13:58 output2/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 4574 2014-05-28 13:58 output2/part-r-00000


查看统计结果
bin/hadoop fs -cat output2/part-r-00000


删除已生成的结果文件
bin/hadoop fs -rmr output*


运行状态:
 bin/hadoop dfsadmin -report


webl界面
[url]http://192.168.1.22:50075/dataNodeHome.jsp[/url]

hfs文件管理
[url]http://192.168.1.22:50070/dfshealth.html[/url]

[url]http://192.168.1.22:50090/status.jsp[/url]

节点管理
[url]http://192.168.1.22:8042/node[/url]

应用管理
[url]http://192.168.1.22/node:8088[/url]

hadoop缺省端口
http://hsrong.iteye.com/blog/1374734

[url="http://powertech.iteye.com/admin/blogs/2082204"]hadoop2.3集群[/url]
当涉及到大数据应用程序的容器化时,通常需要使用多个容器来构建整个系统。以下是一个示例`docker-compose.yml`文件,其中包含一些常见的大数据相关容器: ```yaml version: &#39;3&#39; services: namenode: image: sequenceiq/hadoop-docker:2.7.1 container_name: namenode ports: - "50070:50070" volumes: - ./hadoop-namenode:/hadoop/dfs/name command: /etc/bootstrap.sh -d namenode datanode: image: sequenceiq/hadoop-docker:2.7.1 container_name: datanode volumes: - ./hadoop-datanode:/hadoop/dfs/data environment: - CORE_CONF_fs_defaultFS=hdfs://namenode:8020 command: /etc/bootstrap.sh -d datanode resourcemanager: image: sequenceiq/hadoop-docker:2.7.1 container_name: resourcemanager ports: - "8088:8088" environment: - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager - YARN_CONF_yarn_resourcemanager_webapp_address=resourcemanager:8088 - YARN_CONF_yarn_resourcemanager_address=resourcemanager:8032 command: /etc/bootstrap.sh -d resourcemanager nodemanager: image: sequenceiq/hadoop-docker:2.7.1 container_name: nodemanager environment: - CORE_CONF_fs_defaultFS=hdfs://namenode:8020 - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager command: /etc/bootstrap.sh -d nodemanager historyserver: image: sequenceiq/hadoop-docker:2.7.1 container_name: historyserver environment: - CORE_CONF_fs_defaultFS=hdfs://namenode:8020 - YARN_CONF_yarn_resourcemanager_hostname=resourcemanager command: /etc/bootstrap.sh -d historyserver spark-master: image: bde2020/spark-master:3.1.1-hadoop3.2 container_name: spark-master ports: - "8080:8080" environment: - SPARK_MODE=master command: /bin/bash entrypoint.sh spark-worker: image: bde2020/spark-worker:3.1.1-hadoop3.2 container_name: spark-worker environment: - SPARK_MODE=worker - SPARK_MASTER_URL=spark://spark-master:7077 command: /bin/bash entrypoint.sh kafka: image: wurstmeister/kafka:2.12-2.1.0 container_name: kafka ports: - "9092:9092" environment: - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 depends_on: - zookeeper zookeeper: image: wurstmeister/zookeeper:3.4.6 container_name: zookeeper ports: - "2181:2181"
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值