Hadoop native本地库的编译

hadoop源码编译native本地库

---

1. 检查本地库

hadoop checknative -a
19/08/09 22:13:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop:  false 
zlib:    false 
snappy:  false 
lz4:     false 
bzip2:   false 
openssl: false 
19/08/09 22:13:04 INFO util.ExitUtil: Exiting with status 1

二进制安装的hadoop目录的native是空的，什么都没有，所以checknative 都是false

2. 环境准备

文件如下

1.1.3.tar.gz（snappy，github可以下载）
apache-maven-3.0.5-bin.tar.gz     
jdk-7u80-linux-x64.tar.gz	
apache-ant-1.9.13-bin.zip  
hadoop-2.6.0-cdh5.x.x-src.tar.gz  
protobuf-2.5.0.tar.gz	（github可以下载）

2.1 安装系统所需要的依赖

apt-get install build-essential  openssl libtool   zlib1g-dev libssl-dev  libbz2-dev  autoconf   cmake 

2.2 protobuf 安装

tar zxvf protobuf-2.5.0.tar.gz
./configure
# 等很长一段时间
make 
make install 

protoc --version
如果出现如下错误
protoc: error while loading shared libraries: libprotobuf.so.20: cannot open shared object file: No such file or directory

vi /etc/ld.so.conf 
增加如下一行
/usr/local/lib

ldconfig -v

2.3 snappy的安装

tar zxvf 1.1.3.tar.gz		
cd    snappy-1.1.3
./autogen.sh
./configure
make
make install

3.1 编译hadoop源码

cd hadoop-2.6.0-cdh5.x.x
mvn package  -Pdist,native -DskipTests -Dtar 

如果遇到报错，说是ssl handshake failed ,那就用下面的
mvn package  -Pdist,native -DskipTests -Dtar  -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

还不行，就vi pom.xml  ，将repo改成http的

如果有报错，

例如:
[ERROR] Plugin org.apache.avro:avro-maven-plugin:1.7.6-cdh5.x.x or one of its 
dependencies could not be resolved: Failed to read artifact descriptor for 
org.apache.avro:avro-maven-plugin:jar:1.7.6-cdh5.x.x: Could not transfer artifact 
org.apache.avro:avro-maven-plugin:pom:1.7.6-cdh5.x.x from/to cdh.releases.repo 
(https://repository.cloudera.com/content/repositories/cdh-releases-rcs): peer not authenticated -> [Help 1]

这是因为包没有下载下来的原因，可以使用wget下载，然后放入.m2里面对应的目录

wget -c -r -np -k -L -p   https://repository.cloudera.com/cloudera/cdh-releases-rcs/org/apache/avro/avro-compiler/1.7.6-cdh5.x.x/

3.2 检查native

全部编译完成后，native的文件在 /path/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native

ls   /path/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native

libhadoop.a
libhadooppipes.a
libhadoop.so -> libhadoop.so.1.0.0
libhadoop.so.1.0.0
libhadooputils.a
libhdfs.a
libhdfs.so -> libhdfs.so.0.0.0
libhdfs.so.0.0.0
libnativetask.a
libnativetask.so -> libnativetask.so.1.0.0
libnativetask.so.1.0.0

这里面是没有snappy的so文件的

cp /usr/local/lib/libsnappy.so.1.3.0    /path/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native
cd /path/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native
ln -s libsnappy.so.1.3.0  libsnappy.so.1
ln -s libsnappy.so.1.3.0  libsnappy.so

# 检查native

cd /path/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x
bin/hadoop checknative 

19/08/09 22:43:46 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/08/09 22:43:46 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/path/compile-native/hadoop-2.6.0-cdh5.x.x/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /home/path/compile-native/hadoop-2.6.0-cdh5.x.x/hadoop-dist/target/hadoop-2.6.0-cdh5.x.x/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: true /usr/lib/x86_64-linux-gnu/libcrypto.so

可以看到，已经有了，将native打包，放到其他的haoop节点对应目录下，就可以了