[2011-09-21] Hadoop and HBase 配置汇总

最新推荐文章于 2025-12-01 18:34:51 发布

原创最新推荐文章于 2025-12-01 18:34:51 发布 · 210 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#大数据 #java

ICT 专栏收录该内容

9 篇文章

订阅专栏

本文详细介绍了在Hadoop和HBase环境下进行配置的方法，包括master和slave节点的配置，以及如何正确设置HBase的配置文件和class路径，确保程序在Hadoop上正常运行。

Hadoop

conf/masters


hadoop-master

conf/slaves


hadoop-master
hadoop-slave1
hadoop-slave2

conf/hadoop-env.sh


export JAVA_HOME=/home/michael/lib/jdk1.6.0_27

conf/hdfs-site.xml


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
	<name>dfs.replication</name>
	<value>2</value>
</property>
</configuration>

conf/mapred-site.xml


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
	<name>mapred.job.tracker</name>
	<value>hadoop-master:54311</value>
</property>
</configuration>

conf/core-site.xml


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
	<name>fs.default.name</name>
	<value>hdfs://hadoop-master:54310</value>
</property>
</configuration>

HBase

conf/hbase-env.sh


export JAVA_HOME=/home/michael/lib/jdk1.6.0_27
export HBASE_MANAGES_ZK=true

conf/hbase-site.xml


<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
	<name>hbase.rootdir</name>
	<value>hdfs://hadoop-master:54310/hbase</value>
</property>
<property>
	<name>hbase.cluster.distributed</name>
	<value>true</value>
</property>
<property>
	<name>hbase.master</name>
	<value>hdfs://hadoop-master:6000</value>
</property>

<property>
	<name>hbase.zookeeper.quorum</name>
	<value>yq-cl-svr3</value>
</property>
</configuration>

conf/regionservers


hadoop-master
hadoop-slave1
hadoop-slave2

将两个conf都配置好以后，复制到其他机器即可

update 2011-09-22:

为了让我们写的程序能够在hadoop上正常运行，需要将hbase的配置文件和CLASSPATH添加到hadoop的配置中

首先是配置文件


ln -s ../hbase-0.90.3-cdh3u1/conf/hbase-site.xml conf/hbase-site.xml

然后添加hbase的classpath到hadoop
conf/hadoop-env.sh


export JAVA_HOME=/home/michael/lib/jdk1.6.0_27
export HBASE_HOME=/home/michael/cdh/hbase-0.90.3-cdh3u1
export HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath`

很有用的Tip:
bin/hadoop classpath 和 bin/hbase classpath 可以查看相应的classpath