开源云计算技术系列（六）hypertable（hadoop hdfs)-优快云博客

本文详细介绍了使用VirtualBox搭建Ubuntu Server 9.04环境，通过安装所需软件包并配置Hadoop及hypertable，实现hypertable在Hadoop HDFS上的成功部署与验证。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

选择virtualbox建立ubuntu server 904 的虚拟机作为基础环境。

hadoop@hadoop:~$ sudo apt-get install g++ cmake libboost-dev liblog4cpp5-dev git-core cronolog libgoogle-perftools-dev libevent-dev zlib1g-dev libexpat1-dev libdb4.6++-dev libncurses-dev libreadline5-dev

hadoop@hadoop:~/build/hypertable$ sudo apt-get install ant autoconf automake libtool bison flex pkg-config php5 php5-cli ruby-dev libhttp-access2-ruby libbit-vector-perl

hadoop@hadoop:~/build/hypertable$ sudo ln -f -s /bin/bash /bin/sh
[sudo] password for hadoop:

hadoop@hadoop:~$ tar xvzf hyperic-sigar-1.6.3.tar.gz

hadoop@hadoop:~$ sudo cp hyperic-sigar-1.6.3/sigar-bin/include/*.h /usr/local/include/
[sudo] password for hadoop:
hadoop@hadoop:~$ sudo cp hyperic-sigar-1.6.3/sigar-bin/lib/libsigar-x86-linux.so /usr/local/lib/

hadoop@hadoop:~$ sudo ldconfig

hadoop@hadoop:~/build/hypertable$

hadoop@hadoop:~$ wgethttp://hypertable.org/pub/thrift.tgz
–2009-08-17 21:12:14–http://hypertable.org/pub/thrift.tgz
Resolving hypertable.org… 72.51.43.91
Connecting to hypertable.org|72.51.43.91|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 1144224 (1.1M) [application/x-gzip]
Saving to: `thrift.tgz’

100%[======================================>] 1,144,224 20.9K/s in 44s

2009-08-17 21:13:00 (25.3 KB/s) – `thrift.tgz’ saved [1144224/1144224]

hadoop@hadoop:~$ tar xvzf thrift.tgz

hadoop@hadoop:~$ cd thrift/
hadoop@hadoop:~/thrift$ ls
aclocal config.guess contrib lib NEWS ylwrap
aclocal.m4 config.h CONTRIBUTORS LICENSE NOTICE
bootstrap.sh config.hin depcomp ltmain.sh print_version.sh
CHANGES config.sub DISCLAIMER Makefile.am README
cleanup.sh configure doc Makefile.in test
compiler configure.ac install-sh missing tutorial
hadoop@hadoop:~/thrift$./bootstrap.sh
configure.ac: warning: missing AC_PROG_AWK wanted by: test/py/Makefile:80
configure.ac: warning: missing AC_PROG_RANLIB wanted by: test/py/Makefile:151
configure.ac:44: installing `./config.guess’
configure.ac:44: installing `./config.sub’
configure.ac:26: installing `./install-sh’
configure.ac:26: installing `./missing’
compiler/cpp/Makefile.am: installing `./depcomp’
configure.ac: installing `./ylwrap’

hadoop@hadoop:~/thrift$ ./configure

hadoop@hadoop:~/thrift$ make -j 3

hadoop@hadoop:~/thrift$ sudo make install

hadoop@hadoop:~/thrift$ sudo ldconfig

hadoop@hadoop:~$ ls
hypertable-0.9.2.6-alpha-src.tar.gz jdk-6u15-linux-i586.bin
hadoop@hadoop:~$ chmod +x jdk-6u15-linux-i586.bin
hadoop@hadoop:~$ pwd
/home/hadoop

hadoop@hadoop:~$ ./jdk-6u15-linux-i586.bin

hadoop@hadoop:~$ ls
hypertable-0.9.2.6-alpha-src.tar.gz jdk1.6.0_15 jdk-6u15-linux-i586.bin
hadoop@hadoop:~$ tar xvzf hypertable-0.9.2.6-alpha-src.tar.gz

Create an install directory (optional)

mkdir ~/hypertable

Create a build directory

mkdir -p ~/build/hypertable

hadoop@hadoop:~$cd ~/build/hypertable
hadoop@hadoop:~/build/hypertable$
hadoop@hadoop:~/build/hypertable$ cmake ~/hypertable
hypertable/ hypertable-0.9.2.5-alpha-src.tar.gz
hypertable-0.9.2.5-alpha/
hadoop@hadoop:~/build/hypertable$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/hypertable ~/hypertable-0.9.2.5-alpha

hadoop@hadoop:~/build/hypertable$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/hypertable ~/hypertable-0.9.2.5-alpha

hadoop@hadoop:~/build/hypertable$ make -j 3

这一步时间比较长，并且机器满负载运行，可以喝杯咖啡在来看结果。

hadoop@hadoop:~/build/hypertable$ make install

接下来验证安装是否成功：

hadoop@hadoop:~/build/hypertable$ make alltests
Scanning dependencies of target runtestservers

Starting test servers
Shutdown hyperspace complete
ERROR: DfsBroker not running, database not cleaned
Shutdown thrift broker complete
Shutdown hypertable master complete
Shutdown range server complete
DFS broker: available file descriptors: 1024
Successfully started DFSBroker (local)
Successfully started Hyperspace
Successfully started Hypertable.Master
Successfully started Hypertable.RangeServer
Successfully started ThriftBroker
Built target runtestservers
Scanning dependencies of target alltests
Running tests…
Start processing tests
Test project /home/hadoop/build/hypertable
1/ 60 Testing Common-Exception Passed
2/ 60 Testing Common-Logging Passed
3/ 60 Testing Common-Serialization Passed
4/ 60 Testing Common-ScopeGuard Passed
5/ 60 Testing Common-InetAddr Passed
6/ 60 Testing Common-PageArena Passed
7/ 60 Testing Common-Properties Passed
8/ 60 Testing Common-BloomFilter Passed
9/ 60 Testing Common-Hash Passed
10/ 60 Testing HyperComm Passed
11/ 60 Testing HyperComm-datagram Passed
12/ 60 Testing HyperComm-timeout Passed
13/ 60 Testing HyperComm-timer Passed
14/ 60 Testing HyperComm-reverse-request Passed
15/ 60 Testing BerkeleyDbFilesystem Passed
16/ 60 Testing FileBlockCache Passed
17/ 60 Testing TableIdCache Passed
18/ 60 Testing CellStoreScanner Passed
19/ 60 Testing CellStoreScanner-delete Passed
20/ 60 Testing Schema Passed
21/ 60 Testing LocationCache Passed
22/ 60 Testing LoadDataSource Passed
23/ 60 Testing LoadDataEscape Passed
24/ 60 Testing BlockCompressor-BMZ Passed
25/ 60 Testing BlockCompressor-LZO Passed
26/ 60 Testing BlockCompressor-NONE Passed
27/ 60 Testing BlockCompressor-QUICKLZ Passed
28/ 60 Testing BlockCompressor-ZLIB Passed
29/ 60 Testing CommitLog Passed
30/ 60 Testing MetaLog-RangeServer Passed
31/ 60 Testing Client-large-block Passed
32/ 60 Testing Client-periodic-flush Passed
33/ 60 Testing HyperDfsBroker Passed
34/ 60 Testing Hyperspace Passed
35/ 60 Testing Hypertable-shell Passed
36/ 60 Testing Hypertable-shell-ldi-stdin Passed
37/ 60 Testing RangeServer Passed
38/ 60 Testing ThriftClient-cpp Passed
39/ 60 Testing ThriftClient-perl Passed
40/ 60 Testing ThriftClient-java Passed
41/ 60 Testing Client-random-write-read Passed
42/ 60 Testing RangeServer-commit-log-gc Passed
43/ 60 Testing RangeServer-load-exception Passed
44/ 60 Testing RangeServer-metadata-split Passed
45/ 60 Testing RangeServer-maintenance-thread Passed
46/ 60 Testing RangeServer-row-overflow Passed
47/ 60 Testing RangeServer-rowkey-ag-imbalanc Passed
48/ 60 Testing RangeServer-split-recovery-1 Passed
49/ 60 Testing RangeServer-split-recovery-2 Passed
50/ 60 Testing RangeServer-split-recovery-3 Passed
51/ 60 Testing RangeServer-split-recovery-4 Passed
52/ 60 Testing RangeServer-split-recovery-5 Passed
53/ 60 Testing RangeServer-split-recovery-6 Passed
54/ 60 Testing RangeServer-split-recovery-7 Passed
55/ 60 Testing RangeServer-split-recovery-8 Passed
56/ 60 Testing RangeServer-split-merge-loop10 Passed
57/ 60 Testing RangeServer-bloomfilter-rows Passed
58/ 60 Testing RangeServer-bloomfilter-rows-c Passed
59/ 60 Testing RangeServer-ScanLimit Passed
60/ 60 Testing Client-no-log-sync Passed

100% tests passed, 0 tests failed out of 60
Built target alltests
hadoop@hadoop:~/build/hypertable$

至此，hypertable单机版ok。

接着，我们来体验一下架在hadoop的hdfs上的hypertable。

hadoop@hadoop:~/hadoop-0.20.0/conf$ more hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes.

# The java implementation to use. Required.
export JAVA_HOME=~/jdk1.6.0_15

# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=

# The maximum amount of heap to use, in MB. Default is 1000.
export HADOOP_HEAPSIZE=100

hadoop@hadoop:~/hadoop-0.20.0/conf$ vi core-site.xml
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-0.20.0/tmp/dir/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>

</configuration>

hadoop@hadoop:~/hadoop-0.20.0/conf$ vi mapred-site.xml
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
<description>The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.
</description>
</property>

</configuration>
~

~
“mapred-site.

hadoop@hadoop:~/hadoop-0.20.0/conf$ vi hdfs-site.xml
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>

<!– Put site-specific property overrides in this file. –>

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>

</configuration>
~
~

hadoop@hadoop:~$ more .bash_profile export JAVA_HOME=~/jdk1.6.0_15 export HADOOP_HOME=~/hadoop-0.20.0 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

hadoop@hadoop:~$ hadoop namenode -format
09/08/18 21:10:25 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = hadoop/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.0
STARTUP_MSG: build =https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20-r 763504; compiled by ‘ndaley’ on Thu Apr 9 05:18:40 UTC 2009
************************************************************/
09/08/18 21:10:26 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop,adm,dialout,cdrom,plugdev,lpadmin,sambashare,admin
09/08/18 21:10:26 INFO namenode.FSNamesystem: supergroup=supergroup
09/08/18 21:10:26 INFO namenode.FSNamesystem: isPermissionEnabled=true
09/08/18 21:10:26 INFO common.Storage: Image file of size 96 saved in 0 seconds.
09/08/18 21:10:26 INFO common.Storage: Storage directory /home/hadoop/hadoop-0.20.0/tmp/dir/hadoop-hadoop/dfs/name has been successfully formatted.
09/08/18 21:10:26 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop/127.0.1.1
************************************************************/
hadoop@hadoop:~$

hadoop@hadoop:~$ start-all.sh
starting namenode, logging to /home/hadoop/hadoop-0.20.0/bin/../logs/hadoop-hadoop-namenode-hadoop.out
localhost: starting datanode, logging to /home/hadoop/hadoop-0.20.0/bin/../logs/hadoop-hadoop-datanode-hadoop.out
localhost: starting secondarynamenode, logging to /home/hadoop/hadoop-0.20.0/bin/../logs/hadoop-hadoop-secondarynamenode-hadoop.out
starting jobtracker, logging to /home/hadoop/hadoop-0.20.0/bin/../logs/hadoop-hadoop-jobtracker-hadoop.out
localhost: starting tasktracker, logging to /home/hadoop/hadoop-0.20.0/bin/../logs/hadoop-hadoop-tasktracker-hadoop.out

hadoop@hadoop:~$ jps
12959 JobTracker
12760 DataNode
12657 NameNode
13069 TaskTracker
13149 Jps
12876 SecondaryNameNode

ok，hadoop 0.20.0配置完成。

接着我们整合hypertable和hadoop hdfs。

hadoop@hadoop:~/hypertable/0.9.2.5/bin$ more ~/.bash_profile export JAVA_HOME=~/jdk1.6.0_15 export HADOOP_HOME=~/hadoop-0.20.0 export HYPERTABLE_HOME=~/hypertable/0.9.2.5/ export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HYPERTABLE_HOME/bin

hadoop@hadoop:~/hypertable/0.9.2.5/conf$ ls
hypertable.cfg METADATA.xml
hadoop@hadoop:~/hypertable/0.9.2.5/conf$ more hypertable.cfg
#
# hypertable.cfg
#

# Global properties
Hypertable.Request.Timeout=180000

# HDFS Broker
HdfsBroker.Port=38030
HdfsBroker.fs.default.name=hdfs://localhost:9000
HdfsBroker.Workers=20

# Local Broker
DfsBroker.Local.Port=38030
DfsBroker.Local.Root=fs/local

# DFS Broker – for clients
DfsBroker.Host=localhost
DfsBroker.Port=38030

# Hyperspace
Hyperspace.Master.Host=localhost
Hyperspace.Master.Port=38040
Hyperspace.Master.Dir=hyperspace
Hyperspace.Master.Workers=20

# Hypertable.Master
Hypertable.Master.Host=localhost
Hypertable.Master.Port=38050
Hypertable.Master.Workers=20

# Hypertable.RangeServer
Hypertable.RangeServer.Port=38060

Hyperspace.KeepAlive.Interval=30000
Hyperspace.Lease.Interval=1000000
Hyperspace.GracePeriod=200000

# ThriftBroker
ThriftBroker.Port=38080
hadoop@hadoop:~/hypertable/0.9.2.5/conf$

启动基于hadoop的hypertalbe

hadoop@hadoop:~/hypertable/0.9.2.5/bin$ start-all-servers.sh hadoop
DFS broker: available file descriptors: 1024
Successfully started DFSBroker (hadoop)
Successfully started Hyperspace
Successfully started Hypertable.Master
Successfully started Hypertable.RangeServer
Successfully started ThriftBroker

hadoop@hadoop:~/hypertable/0.9.2.5/log$ hadoop fs -ls /
Found 2 items
drwxr-xr-x – hadoop supergroup 0 2009-08-18 21:25 /home
drwxr-xr-x – hadoop supergroup 0 2009-08-18 21:28 /hypertable
hadoop@hadoop:~/hypertable/0.9.2.5/log$ hadoop fs -ls /hypertable
Found 2 items
drwxr-xr-x – hadoop supergroup 0 2009-08-18 21:28 /hypertable/server
drwxr-xr-x – hadoop supergroup 0 2009-08-18 21:28 /hypertable/tables

至此，基于hadoop的hypertable成功启动。下一篇进行hql的体验。

转自：http://rdc.taobao.com/blog/dw/archives/430