HBase配置snappy

最新推荐文章于 2023-09-11 10:41:07 发布

迷途小码

最新推荐文章于 2023-09-11 10:41:07 发布

阅读量1.5k

点赞数

分类专栏： hbase安装&运行

hbase安装&运行专栏收录该内容

5 篇文章

订阅专栏

refer:

http://louishust.github.io/hbase/2015/07/14/snappy-install

编译安装snappy

git clone https://github.com/google/snappy.git
yum install -y automake autoconf gcc-c++ cmake libedit
./configure
make
make install

编译安装hadoop-snappy

git clone https://github.com/louishust/hadoop-snappy.git
ln -s /home/q/java/default/jre/lib/amd64/server/libjvm.so /usr/local/lib/
mvn package
cp target/hadoop-snappy-0.0.1-SNAPSHOT.jar /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/
cp target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/*  /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native
cp  target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* /home/q/hbase/hbase-0.98.6-cdh5.2.0/lib/native/Linux-amd64-64/

重新编译libhadoop以支持snappy

首先需要protocbuf 2.5.0以上

mvn clean package -Pdist,native -DskipTests -Drequire.snappy
cp  /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/src/hadoop-dist/target/hadoop-2.5.0-cdh5.2.0/lib/native/* /home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native

修改hadoop配置文件

core-site.xml

<property>
     <name>io.compression.codecs</name>
     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

hadoop-env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native:/usr/local/lib/
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib/native:/home/q/hadoop/hadoop-2.5.0-cdh5.2.0/lib

yarn-site.xml

<property>
     <name>mapreduce.map.output.compress</name>
     <value>true</value>
</property>
<property>
     <name>mapred.map.output.compress.codec</name>
     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

修改hbase配置文件

hbase-env.sh

### for snappyexport HBASE_HOME=/home/q/hbase/hbase-0.98.6-cdh5.2.0/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/:/usr/local/lib/
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/Linux-amd64-64/:/usr/local/lib/
export CLASSPATH=$CLASSPATH:$HBASE_LIBRARY_PATH

配置libjvm

sudo ln -s /home/q/java/default/jre/lib/amd64/server/libjvm.so /usr/local/lib/

重启集群

bin/stop-hbase.sh
sbin/stop-dfs.sh
sbin/start-dfs.sh
bin/start-hbase.sh

验证snappy

touch /tmp/a
hbase org.apache.hadoop.hbase.util.CompressionTest /tmp/a snappy
bin/hbase shell
create 'tsnappy',{ NAME => 'f', COMPRESSION => 'snappy'}

转载请注明出处：Louis Hust's Blog