hadoop-2.7.6+snappy

最新推荐文章于 2018-12-12 19:14:18 发布

Sanshierli_

最新推荐文章于 2018-12-12 19:14:18 发布

阅读量531

点赞数

分类专栏：大数据

本文链接：https://blog.youkuaiyun.com/qq_40931181/article/details/83615896

版权

大数据专栏收录该内容

1 篇文章

订阅专栏

记录一下snappy安装过程

环境配置：hadoop-2.7.6 Maven 3.5.4 JDK1.8 HBase1.4.4 gcc5.4.0

1. Requirements

sudo apt install autoconf
sudo apt install automake
sudo apt install libtool-bin
sudo apt install cmake

2. protobuf

tar -zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/protobuf-2.5.0/ 
make
sudo make install
export LD_LIBRARY_PATH=/usr/local/lib
sudo ldconfig

检验安装结果

~$ protoc --version
libprotoc 2.5.0

3. 安装snappy

tar -zxvf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make
sudo make install

ll /usr/local/lib/ | grep snappy

4. 编译hadoop

sudo gedit /etc/profile
export HADOOP_OPTS="-Djava.library.path=${HADOOP_HOME}/lib/native"
source /etc/profile

tar -zxvf hadoop-2.7.6-src.tar.gz
cd hadoop-2.7.6-src
mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy

hadoop-dist/target/hadoop------/lib/native 将此目录下的文件拷贝到hadoop集群中的hadoop下的lib/native目录和hbase下的lib/native/Linux-amd64-64目录下，没有则新建，各节点均需拷贝

core-site.xml
<property>
        <name>io.compression.codecs</name>
        <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

hbase-site.xml
<property>
        <name>hbase.block.data.cachecompressed</name>
        <value>true</value>
</property>

snappy 压缩

https://blog.youkuaiyun.com/maomaosi2009/article/details/47019913

https://www.cnblogs.com/shitouer/archive/2013/01/14/2859475.html

https://blog.youkuaiyun.com/chenyi8888/article/details/14167311

mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy