Hadoop3.1.3 + Hbase2.1.7 设置Snappy压缩算法

一、查看Linux是否有系统自带的snappy库，如果有删除掉自带的snappy库

① 查看Linux是否有系统自带的snappy库

ll /usr/lib64 | grep snappy

② 删除掉自带的snappy库

yum -y remove snappy 

二、安装snappy本地库

① 下载snappy：

wget https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.4.tar.gz/sha512/873f655713611f4bdfc13ab2a6d09245681f427fbd4f6a7a880a49b8c526875dbdd623e203905450268f542be24a2dc9dae50e6acc1516af1d2ffff3f96553da/snappy-1.1.4.tar.gz

② 安装

tar zxvf snappy-1.1.4.tar.gz -C /usr/local/snappy
cd /usr/local/snappy/snappy-1.1.4
./autogen.sh
./configure
make
make install

默认安装到/usr/local/lib目录

③ 添加Snappy本地库至/usr/lib64目录下

cp -d /usr/local/lib/* /usr/lib64

三、安装hadoop-snappy

① 安装hadoop-snappy需要一系列的依赖

sudo yum -y install gcc c++ autoconf automake libtool 

② 安装

git clone https://github.com/electrum/hadoop-snappy.gitcd hadoop-snappy/mvn package

四、hadoop配置snappy

① 添加Snappy本地库至$HADOOP_HOME/lib/native/目录下

cp -d /usr/local/lib/* /usr/local/hadoop/hadoop-3.1.3/lib/native

②将hadoop-snappy-0.0.1-SNAPSHOT.jar和snappy的library分别拷贝到 $HADOOP_HOME/lib 和$HADOOP_HOME/lib/native/ 目录下即可

cp /home/hadoop/snappy/hadoop-snappy/target/hadoop-snappy-0.0.1-SNAPSHOT.jar $HADOOP_HOME/lib
cp /home/hadoop/snappy/hadoop-snappy/target/hadoop-snappy-0.0.1-SNAPSHOT-tar/hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/

③ 配置 hadoop-env.sh 和 core-site.xml，mapred-site.xml

vim hadoop-env.sh
export LD_LIBRARY_PATH