Hbase Cluster Setup Guide

Hbase Cluster Setup Guide

Environment:

1.       OS: Ubuntu 12.04.2 LTS

2.       JAVA: JDK1.7

3.       Hadoop: Hadoop 2.0.3 alpha

4.       Maven: Maven3

5.       Hbase: hbase-0.94.8-security.tar.gz

Preparation

1.       unlimit on ubuntu

Set the number of the files all at the same time in memory. Default is 1024.

a.       Add the content at the end of the file /etc/security/limits.conf. hduser is the hadoop and hbase user. Replace this with yours.

hduser – nofile 32768          

b.       Add the content at the end of the file /etc/pam.d/common-session

session required pam_limits.so

 

2.       Install maven3

$ sudo apt-get update

$ sudo apt-get install maven

 

3.       Rebiuld hbase-0.94.8-security

a)         tar the source code and grant the hduser to the folder

$ tar –zxvf hbase-0.94.8-security.tar.gz

$ sudo mv hbase-0.94.8-security /usr/local/hbase94

$ sudo chmod –R hduser:hduser /usr/local/hbase94

b)         Change the hadoop version for the pom.xml, replace 2.0.0-alpha with 2.0.3-alpha shown with red font.

        

                   2.0.3-alpha

        1.6.1

 

 

 

        2.0.0-alpha

        1.6.1

    

c)         Rebuild the hbase0.94 with maven

  $ cd /usr/local/hbase94

  $ mvn clean –Dhadoop.profile=2.0 –DskipTests

  $ mvn compile –Dhadoop.profile=2.0 –DskipTests

  $ mvn package –Dhadoop.profile=2.0 –DskipTests

  $ mvn install –Dhadoop.profile=2.0 –DskipTests

d)        Copy the file under /usr/local/hbase0.94/target/ hbase-0.94.8.tar.gz. Now we will use the recompiled hbase with hadoop2.0.3 to set up our hbase cluster

4.       NTP (Networking Time Protocol)

$ sudo atp-get install ntp

$ sudo service ntp stop

$ sudo ntpdate –u ntp.ubuntu.com

$ sudo service ntp start

 

5.       dfs.datanode.max.xcievers

Configure the property for hadoop in hdfs-site.xml

 

        dfs.datanode.max.xcievers

        4096

     

configuration

Standalone HBASE

Don’t use HDFS to store the database files. It uses local file system.

$ tar –zxvf hbase-0.94.8.tar.gz

$sudo mv hbase-0.94.8 /usr/local/ hbase-0.94.8

$sudo chmod –R hduser:hduser /usr/local/ hbase-0.94.8

Make directory for hbase.root and hbase.zookeeper.property.dataDir

$sudo mkdir /home/hbase

$sudo mkdir /home/zookeeper

$sudo chmod –R hduser:hduser /home/hbase

$sudo chmod –R hduser:hduser /home/zookeeper

Edit /usr/local/ hbase-0.94.8/conf/hbase-site.xml. Set hbase.root and hbase.zookeeper.property.dataDir

 

 

 

 

    hbase.rootdir

    file:///home/hbase

 

 

    hbase.zookeeper.property.dataDir

    /home/zookeeper

 

 

Pseudo-distributed

A pseudo-distributed mode is simply a distributed mode run on a single host. Use this configuration testing and prototyping on HBase. Do not use this configuration for production nor for evaluating HBase performance. Here will not provide the way to configure it.

Fully-distributed

1 regionserver

         i.            tar the file hbase-0.94.8-security.tar.gz

$ tar –zxvf hbase-0.94.8.tar.gz

$sudo mv hbase-0.94.8 /usr/local/hbase-0.94.8

$sudo chmod –R hduser:hduser /usr/local/hbase-0.94.84

 

       ii.            Make dir for the hbase in hadoop file system.

$ /user/local/hadoop/bin/hdfs dfs –mkdir hdfs://master:8888/hbase

 

      iii.            HDFS Client Configuration

a.       Let Hbase know the configuration of the hadoop, do one of the operations below. Here I choose the second operation.

u  Add a pointer to your HADOOP_CONF_DIR to the HBASE_CLASSPATH environment variable in hbase-env.sh.

u  Add a copy of hdfs-site.xml or, better, symlinks, under ${HBASE_HOME}/conf, or

u  if only a small set of HDFS client configurations, add them to hbase-site.xml.

b.       Set the upper bound on the number of files that it will serve at any one time. Configure hadoop’s etc/hadoop/hdfs-site.xml setting the xceivers value to at least the following:

 

        dfs.datanode.max.xcievers

        4096

     

Restart hadoop to source it.

 

     iv.            Configure the file conf/hbase-site.xml

$ cd /usr/local/hbase-0.94.8

$ vi conf/hbase-site.xml

 

 

       

                hbase.rootdir

                hdfs://master:8888/hbase

       

 

                hbase.zookeeper.quorum

                master

       

       

                hbase.zookeeper.property.clientport

                2222

       

       

       

                hbase.zookeeper.property.datadir

                /home/zookeeper

       

       

                hbase.cluster.distributed

                true

       

 

 

       v.            Configure regionservers add the regionservers’ hostnames.

$ vi conf/regionservers

master

 

     vi.            Configure JAVA_HOME in conf/hbase-env.sh, add below content

export JAVA_HOME = /usr/local/jdk1.7

 

2 regionservers

Master:

       1.            Set up as 1 regionserver set

       2.            Configure regionservers and add the regionservers’ hostnames

$ vi conf/regionservers

master

slave

       3.            Configure the file conf/hbase-site.xml

 

       

                hbase.rootdir

                hdfs://master:8888/hbase

       

 

                hbase.zookeeper.quorum

                master, salve

       

       

                hbase.zookeeper.property.clientport

                2222

       

       

       

                hbase.zookeeper.property.datadir

                /home/zookeeper

       

       

                hbase.cluster.distributed

                true

       

 

 

Slave

1.       tar the file hbase-0.94.8-security.tar.gz

$ tar –zxvf hbase-0.94.8.tar.gz

$sudo mv hbase-0.94.8 /usr/local/hbase-0.94.8

$sudo chmod –R hduser:hduser /usr/local/hbase-0.94.84

 

2.       Configure regionservers add the regionservers’ hostnames

$ vi conf/regionserver

           master

           slave

 

3.       Configure the file conf/hbase-site.xml

 

       

                hbase.rootdir

                hdfs://master:8888/hbase

       

 

                hbase.zookeeper.quorum

                master, salve

       

       

                hbase.zookeeper.property.clientport

                2222

       

       

       

                hbase.zookeeper.property.datadir

                /home/zookeeper

       

       

                hbase.cluster.distributed

                true

       

 

 

 

Start/Stop hbase

1.       Start

$ bin/start-hbase.sh

$ jps

3 services will be added in java process:

         12457 HQuorumPeer

         12610 HRegionServer

         12514 HMaster

 

2.       Stop

$ bin/stop-hbase.sh

 

Practice Hbase

1.       Use hbase shell

$ bin/hbase shell

 

2.       Create table

hbase(main) :001:0> create ‘test’,’cf’

 

3.       Insert records

hbase(main) :001:0> put 'test', 'row1', 'cf:a', 'value1'

hbase(main) :001:0> put 'test', 'row2', 'cf:b', 'value2'

hbase(main) :001:0> put 'test', 'row3', 'cf:c', 'value3'

 

4.       Scan the table

hbase(main) :001:0>  scan 'test'

ROW        COLUMN+CELL

row1       column=cf:a, timestamp=1288380727188, value=value1

row2       column=cf:b, timestamp=1288380738440, value=value2

row3       column=cf:c, timestamp=1288380747365, value=value3

3 row(s) in 0.0590 seconds

 

5.       disable the table and drop it

hbase(main):012:0> disable 'test'

0 row(s) in 1.0930 seconds

hbase(main):013:0> drop 'test'

0 row(s) in 0.0770 seconds