转载:
https://blog.youkuaiyun.com/qq_32641659/article/details/89337897
1、文件压缩配置实现
首先你的Hadoop是需要编译安装的,参考博客:Hadoop源码编译
https://blog.youkuaiyun.com/greenplum_xiaofan/article/details/95466703
检查Hadoop支持的压缩格式:
[hadoop@vm01 hadoop-2.6.0-cdh5.7.0]$ pwd
/home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0
[hadoop@vm01 hadoop-2.6.0-cdh5.7.0]$ ./bin/hadoop checknative
19/07/10 23:35:35 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/07/10 23:35:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/hadoop/source/hadoop-2.6.0-cdh5.7.0/hadoop-dist/target/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
hadoop checknative 虽然没没显示gzip、LZO压缩格式是否支持,是因为检查的是native,只要本机有gzip和LZO相关软件即可
2、修改core-site.xml
参数支持压缩
<property>
<name