编译apache hadoop3.0.2

以下编译基于CentOS 7.3 环境：

0、软件准备

• JDK1.8
• apache-maven-3.6.1(needs 3.3+)
• hadoop-3.0.2-src 源码（下载地址：https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.0.2/hadoop-3.0.2-src.tar.gz）
• protobuf-2.5.0 源码编译（下载地址：https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz）

1、编译环境准备

1.1 maven配置

maven添加aliyun等国内镜像源，否则下载速度极慢，严重影响编译时的心情。

<mirror>
        <id>nexus-aliyun</id>
        <mirrorOf>*</mirrorOf>
        <name>Nexus aliyun</name>
        <url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

1.2 protobuf编译安装

hadoop源码使用了protobuf2.5.0版本的库包，需要事先对protobuf2.5.0进行编译：

./configure --prefix=/usr/local/protobuf
make && make install

检验是否安装成功：

protoc --version
libprotoc 2.5.0

1.3 CMake3安装

yum -y install cmake3 
ln -s /usr/bin/cmake3 /usr/bin/cmake

1.4 编译aws模块找不到依赖Jar包

在编译“Amazon Web Service Support”模块时，提示找不到DynamoDBLocal.jar包

一种可行的解决办法是：

mkdir -p ${M2_HOME}/repository/com/amazonaws/DynamoDBLocal/1.11.86

再将离线下载好的DynamoDBLocal-1.11.86.jar文件拷贝至该目录下

2、开始编译

2.1 编译命令

解压后的hadoop-3.0.2-src源码目录下，执行如下maven命令开始编译：

mvn package -Pnative -Drequire.fuse=true -DskipTests -Dmaven.javadoc.skip=true

参数说明：

1.-P,–activate-profiles Comma-delimited list of profiles to activate
2.require.fuse=true， fuse-dfs executable can be built by setting require.fuse option to true using Maven
3. 忽略测试、忽略构建javadoc

编译详细信息：
https://github.com/apache/hadoop/tree/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/fuse-dfs/doc

2.2 编译结果

上述命令键入后，等待编译完毕，期间如果遇到问题，按提示解决即可。
3.0.2版本版本编译主要遇到上述几个问题，详细错误日志，详见附件图片。

编译完成后的summary日志：

[INFO] Executed tasks
[INFO]
[INFO] — maven-remote-resources-plugin:1.5:process (default) @ hadoop-cloud-storage-project —
[INFO]
[INFO] — maven-site-plugin:3.6:attach-descriptor (attach-descriptor) @ hadoop-cloud-storage-project —
[INFO] No site descriptor found: nothing to attach.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.0.2:
[INFO]
[INFO] Apache Hadoop Main … SUCCESS [ 1.208 s]
[INFO] Apache Hadoop Build Tools … SUCCESS [ 1.103 s]
[INFO] Apache Hadoop Project POM … SUCCESS [ 0.454 s]
[INFO] Apache Hadoop Annotations … SUCCESS [ 0.375 s]
[INFO] Apache Hadoop Project Dist POM … SUCCESS [ 0.063 s]
[INFO] Apache Hadoop Assemblies … SUCCESS [ 0.066 s]
[INFO] Apache Hadoop Maven Plugins … SUCCESS [ 2.054 s]
[INFO] Apache Hadoop MiniKDC … SUCCESS [ 0.300 s]
[INFO] Apache Hadoop Auth … SUCCESS [ 2.263 s]
[INFO] Apache Hadoop Auth Examples … SUCCESS [ 0.840 s]
[INFO] Apache Hadoop Common … SUCCESS [ 8.631 s]
[INFO] Apache Hadoop NFS … SUCCESS [ 0.850 s]
[INFO] Apache Hadoop KMS … SUCCESS [ 0.933 s]
[INFO] Apache Hadoop Common Project … SUCCESS [ 0.032 s]
[INFO] Apache Hadoop HDFS Client … SUCCESS [ 1.761 s]
[INFO] Apache Hadoop HDFS … SUCCESS [ 2.233 s]
[INFO] Apache Hadoop HDFS Native Client … SUCCESS [ 0.570 s]
[INFO] Apache Hadoop HttpFS … SUCCESS [ 1.049 s]
[INFO] Apache Hadoop HDFS-NFS … SUCCESS [ 0.301 s]
[INFO] Apache Hadoop HDFS Project … SUCCESS [ 0.022 s]
[INFO] Apache Hadoop YARN … SUCCESS [ 0.023 s]
[INFO] Apache Hadoop YARN API … SUCCESS [ 0.567 s]
[INFO] Apache Hadoop YARN Common … SUCCESS [ 1.137 s]
[INFO] Apache Hadoop YARN Server … SUCCESS [ 0.022 s]
[INFO] Apache Hadoop YARN Server Common … SUCCESS [ 0.786 s]
[INFO] Apache Hadoop YARN Registry … SUCCESS [ 0.833 s]
[INFO] Apache Hadoop YARN NodeManager … SUCCESS [ 1.040 s]
[INFO] Apache Hadoop YARN Web Proxy … SUCCESS [ 0.429 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService … SUCCESS [ 0.538 s]
[INFO] Apache Hadoop YARN Timeline Service … SUCCESS [ 0.628 s]
[INFO] Apache Hadoop YARN ResourceManager … SUCCESS [ 1.870 s]
[INFO] Apache Hadoop YARN Server Tests … SUCCESS [ 0.734 s]
[INFO] Apache Hadoop YARN Client … SUCCESS [ 0.478 s]
[INFO] Apache Hadoop YARN SharedCacheManager … SUCCESS [ 0.662 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage … SUCCESS [ 0.919 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend … SUCCESS [ 1.384 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests … SUCCESS [ 2.039 s]
[INFO] Apache Hadoop YARN Router … SUCCESS [ 0.723 s]
[INFO] Apache Hadoop YARN Applications … SUCCESS [ 0.022 s]
[INFO] Apache Hadoop YARN DistributedShell … SUCCESS [ 0.590 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher … SUCCESS [ 0.410 s]
[INFO] Apache Hadoop YARN Site … SUCCESS [ 0.025 s]
[INFO] Apache Hadoop YARN UI … SUCCESS [ 0.026 s]
[INFO] Apache Hadoop YARN Project … SUCCESS [ 1.173 s]
[INFO] Apache Hadoop MapReduce Client … SUCCESS [ 0.095 s]
[INFO] Apache Hadoop MapReduce Core … SUCCESS [ 0.999 s]
[INFO] Apache Hadoop MapReduce Common … SUCCESS [ 0.627 s]
[INFO] Apache Hadoop MapReduce Shuffle … SUCCESS [ 0.738 s]
[INFO] Apache Hadoop MapReduce App … SUCCESS [ 0.735 s]
[INFO] Apache Hadoop MapReduce HistoryServer … SUCCESS [ 0.901 s]
[INFO] Apache Hadoop MapReduce JobClient … SUCCESS [ 0.694 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins … SUCCESS [ 0.335 s]
[INFO] Apache Hadoop MapReduce NativeTask … SUCCESS [ 1.506 s]
[INFO] Apache Hadoop MapReduce Examples … SUCCESS [ 0.458 s]
[INFO] Apache Hadoop MapReduce … SUCCESS [ 0.122 s]
[INFO] Apache Hadoop MapReduce Streaming … SUCCESS [ 0.481 s]
[INFO] Apache Hadoop Distributed Copy … SUCCESS [ 0.332 s]
[INFO] Apache Hadoop Archives … SUCCESS [ 0.129 s]
[INFO] Apache Hadoop Archive Logs … SUCCESS [ 0.272 s]
[INFO] Apache Hadoop Rumen … SUCCESS [ 0.300 s]
[INFO] Apache Hadoop Gridmix … SUCCESS [ 0.163 s]
[INFO] Apache Hadoop Data Join … SUCCESS [ 0.115 s]
[INFO] Apache Hadoop Extras … SUCCESS [ 0.193 s]
[INFO] Apache Hadoop Pipes … SUCCESS [ 0.173 s]
[INFO] Apache Hadoop OpenStack support … SUCCESS [ 0.466 s]
[INFO] Apache Hadoop Amazon Web Services support … SUCCESS [ 2.450 s]
[INFO] Apache Hadoop Kafka Library support … SUCCESS [ 3.194 s]
[INFO] Apache Hadoop Azure support … SUCCESS [ 3.866 s]
[INFO] Apache Hadoop Aliyun OSS support … SUCCESS [ 1.531 s]
[INFO] Apache Hadoop Client Aggregator … SUCCESS [ 1.402 s]
[INFO] Apache Hadoop Mini-Cluster … SUCCESS [ 1.224 s]
[INFO] Apache Hadoop Scheduler Load Simulator … SUCCESS [ 1.788 s]
[INFO] Apache Hadoop Resource Estimator Service … SUCCESS [ 1.885 s]
[INFO] Apache Hadoop Azure Data Lake support … SUCCESS [ 2.471 s]
[INFO] Apache Hadoop Tools Dist … SUCCESS [ 0.969 s]
[INFO] Apache Hadoop Tools … SUCCESS [ 0.021 s]
[INFO] Apache Hadoop Client API … SUCCESS [01:23 min]
[INFO] Apache Hadoop Client Runtime … SUCCESS [01:05 min]
[INFO] Apache Hadoop Client Packaging Invariants … SUCCESS [ 1.790 s]
[INFO] Apache Hadoop Client Test Minicluster … SUCCESS [01:29 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.138 s]
[INFO] Apache Hadoop Client Packaging Integration Tests … SUCCESS [ 0.070 s]
[INFO] Apache Hadoop Distribution … SUCCESS [ 2.310 s]
[INFO] Apache Hadoop Client Modules … SUCCESS [ 0.021 s]
[INFO] Apache Hadoop Cloud Storage … SUCCESS [ 0.777 s]
[INFO] Apache Hadoop Cloud Storage Project … SUCCESS [ 0.028 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:18 min
[INFO] Finished at: 2019-06-25T16:29:17+08:00
[INFO] ------------------------------------------------------------------------

3. 编译报错截图

3.1 google protobuf序列化库未找到

3.2 CMake3编译工具未找到

3.3 Amazon aws DynamoDBLocal.jar未找到