Hadoop添加snappy压缩支持

最新推荐文章于 2021-12-04 13:52:11 发布

IT-Billy

最新推荐文章于 2021-12-04 13:52:11 发布

阅读量1.4k

点赞数 2

文章标签： hadoop snappy

本文链接：https://blog.youkuaiyun.com/Python_Billy/article/details/115168522

版权

本文介绍了如何在Hadoop环境中添加Snappy压缩支持。当使用`hadoop checknative`命令发现Snappy为false时，需要手动下载并安装Snappy库。通过下载snappy-1.1.3.tar.gz，解压、编译和安装，然后将库文件复制到Hadoop的native目录，并更新core-site.xml和mapred-site.xml配置文件。重启Hadoop后，Snappy应显示为true，表明配置成功。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

启动hadoop后，使用如下命令查看压缩支持：

hadoop checknative

出现下列输出，snappy为false，则需要手动添加snappy支持

Native library checking:
hadoop:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
snappy:  false 
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.

snappy-1.1.3.tar.gz下载地址：
https://src.fedoraproject.org/lookaside/pkgs/snappy/snappy-1.1.3.tar.gz/7358c82f133dc77798e4c2062a749b73/
下载snappy-1.1.3.tar.gz包，然后执行以下命令

tar -xvf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make
make install

默认目录是 /usr/local/lib，查看snappy是否安装成功

ll /usr/local/lib | grep snappy

出现下面5个文件，表示安装成功

libsnappy.a
libsnappy.la*
libsnappy.so -> libsnappy.so.1.3.0*
libsnappy.so.1 -> libsnappy.so.1.3.0*
libsnappy.so.1.3.0*

将上述5个文件复制到${HADOOP_HOME}/lib/native，${HADOOP_HOME}为你hadoop的安装目录
修改hadoop下的/etc/hadoop/core-site.xml文件，添加以下属性，如果是集群，记得分发

<!-- 压缩相关 -->
<property>
	<name>io.compression.codecs</name>
	<value>
		org.apache.hadoop.io.compress.GzipCodec,
		org.apache.hadoop.io.compress.DefaultCodec,
		org.apache.hadoop.io.compress.BZip2Codec,
		org.apache.hadoop.io.compress.SnappyCodec
	</value>
</property>

修改hadoop下的/etc/hadoop/mapred-site.xml文件，添加以下属性，如果是集群，记得分发

<!-- 压缩相关 -->
<property>
	<name>mapred.output.compress</name>
	<value>true</value>
</property>
<property>
	<name>mapred.output.compression.codec</name>
	<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
	<name>mapred.compress.map.output</name>
	<value>true</value>
</property>
<property>
	<name>mapred.map.output.compression.codec</name>
	<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

重启hadoop集群，再次执行 hadoop checknative，snappy那一栏不是false则表示成功。

例如：

Native library checking:
hadoop:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
snappy:  true /opt/bigdata/hadoop/hadoop-3.1.4/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:    false The native code was built without PMDK support.