Hadoop集群搭建之三 配置文件说明

1.集群软件选择及环境部署

1)集群软件选择

软件	版本	下载地址	安装位置
操作系统	CentOS-7-x86_64-DVD-1611.iso	http://101.96.10.26/isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1611.iso	虚拟机Vmware10
JAVA	jdk-8u131-linux-x64.tar.gz	http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html	/usr/local/hadoopenv/java/jdk1.8.0_131
Hadoop	hadoop-2.7.3.tar.gz	http://apache.fayea.com/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz	/usr/local/hadoopenv/hadoop
Zookeeper	Zookeeper-3.4.8.tar.gz	http://mirrors.hust.edu.cn/apache/zookeeper/	/usr/local/hadoopenv/zookeeper
Hbase	hbase-1.2.6-bin.tar	https://archive.apache.org/dist/hbase/stable/	/usr/local/hadoopenv/hbase

2)单机情况下伪分布式模式下集群主机和ip映射

集群角色	主机名	用户名	用户IP
master	namenode	hadoop
slave1	datanode1	hadoop
slave2	datanode2	hadoop

2.Hadoop1.2.6配置文件说明

1)core-site.xml

name	value
hadoop.tmp.dir	/usr/local/hadoopenv/tmp/hadoop
fs.defaultFs	hdfs://namenode:9000

2)hadoop-env.sh

export  JAVA_HOME=/usr/local/hadoopenv/java/jdk1.8.0_131
export  HADOOP_LOG_DIR= /usr/local/hadoopenv/tmp /hadoop /dfs/logs
export  HADOOP_PID_DIR= /usr/local/hadoopenv/tmp /hadoop /dfs/pids

3)mapred-env.sh

export  JAVA_HOME=/usr/local/hadoopenv/java/jdk1.8.0_131
export  HADOOP_LOG_DIR= /usr/local/hadoopenv/tmp /hadoop /dfs/logs
export  HADOOP_PID_DIR= /usr/local/hadoopenv/tmp /hadoop /dfs/pids

4)yarn-env.sh

export JAVA_HOME=/usr/local/hadoopenv/java/jdk1.8.0_131
export YARN_LOG_DIR= /usr/local/hadoopenv/ tmp/hadoop/ dfs/logs

5)hdfs-site.xml

name	value
dfs.replication	3
dfs.namenode.name.dir	file:/usr/local/hadoopenv/tmp/hadoop/dfs/name
dfs.datanode.name.dir	file:/usr/local/hadoopenv/tmp/hadoop/dfs/data
dfs.namenode.secondary.http-address	namenode:9001
dfs.webhdfs.enabled	true

6)mapred-site.xml

name	value
mapreduce.framework.name	yarn
mapreduce.jok.tracker	localhoost:54311
mapreduce.jobhistory.address	namenode:10020
mapreduce.jobtracker.http.address	namenode:50030
mapreduce.jobhistory.webapp.address	namenode:19888

7)slaves

datanode1
datanode2

8)yarn-site.xml

name	value
yarn.nodemanager.aux-services	mapreduce.shuffle
yarn.resourcemanager.hostname	namenode
yarn.resourcemanager.scheduler.address	namenode:8030
yarn.resourcemanager.resource-tracker.address	namenode:8031
yarn.resourcemanager.address	master:8032
yarn.resourcemanager.admin.address	master:8033
yarn.resourcemanager.webapp.address	master:8088

3.zookeeper3.4.8

1)zoo.cfg

tickTime=6000
initLimit=10
syncLimit=5
dataDir=/usr/local/hadoopenv/tmp/zookeeper/data
server.1=namenode:2888:3888
server.2=datanode1:2888:3888
server.3=datanode2:2888:3888
clientPort=2181

4.Hbase1.2.6

1)hbase-env.sh

export JAVA_HOME=/usr/local/hadoopenv/java/jdk1.8.0_131/
export HBASE_CLASSPATH=/usr/local/hadoopenv/hbase
export HBASE_MANAGES_ZK=false

2)hbase-site.xml

value	name
hdfs://namenode:9000/hbase	hbase.rootdir
namenode:60000	hbase.master
/usr/local/hadoopenv/tmp/zookeeper/data	hbase.zookeeper.property.dataDir
true	hbase.cluster.distributed
namenode,datanode1,datanode2	hbase.zookeeper.quorum
150000	hbase.master.maxclockskew
6000	zookeeper.session.ticktime

3)regionservers

datanode1
datanode2