CentOS下编译hadoop2.7.5

CentOS下编译hadoop2.7.5 

说明：
本文测试环境：虚拟机VMware Workstation Pro、操作系统 CentOS 6.8 64 位
基于上述环境成功编译hadoop2.7.5、hadoop-2.6.5，其它环境或版本请参照编译。
本文以编译hadoop2.7.5的过程作介绍

一、编译前准备
下载hadoop-2.7.5-src.tar.gz (http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.7.5/hadoop-2.7.5-src.tar.gz)
解压hadoop后仔细阅编译指导文件
[root@Master2 soft]# tar -zxvf hadoop-2.7.5-src.tar.gz
[root@Master2 hadoop-2.7.5-src]# less BUILDING.txt
下载准备如下软件
jdk-7u45-linux-x64.tar.gz
apache-ant-1.9.9-bin.tar.gz
apache-maven-3.0.5-bin.tar.gz
findbugs-3.0.1.tar.gz
hadoop-2.7.5-src.tar.gz
protobuf-2.5.0.tar.gz

二、安装JDK
编译要求：JDK 1.7+
安装软件：jdk-7u45-linux-x64.tar.gz

tar -zxvf jdk-7u45-linux-x64.tar.gz -C /usr/local
修改配置文件
export JAVA_HOME=/usr/local/jdk1.7.0_45
export PATH=$PATH:$JAVA_HOME/bin

三、安装MAVEN
编译要求：Maven 3.0 or later
安装软件：apache-maven-3.0.5-bin.tar.gz
[root@Master2 soft]# tar -zxvf apache-maven-3.0.5-bin.tar.gz -C /opt
修改配置文件
export M2_HOME=/opt/apache-maven-3.0.5
export PATH=$PATH:$M2_HOME/bin
测试：
[root@Master2 ~]# mvn -v
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /opt/apache-maven-3.0.5
Java version: 1.7.0_45, vendor: Oracle Corporation
Java home: /usr/local/jdk1.7.0_45/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.el6.x86_64", arch: "amd64", family: "unix"

四、安装findbugs
编译要求：Findbugs 1.3.9 
安装软件：findbugs-3.0.1.tar.gz
[root@Master2 soft]# tar -zxvf findbugs-3.0.1.tar.gz -C /opt
修改配置文件
[root@Master2 ~]# vim /etc/profile
export FINDBUGS_HOME=/opt/findbugs-3.0.1
export PATH=$PATH:$FINDBUGS_HOME/bin
测试：
[root@Master2 ~]# findbugs -version

3.0.1

五、安装依赖包

根据编译指导文件BUILDING.txt，安装相关依赖程序包
[root@Master2 protobuf-2.5.0]# yum -y install gcc-c++ build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-devua
[root@Master2 protobuf-2.5.0]# rpm -q gcc-c++  build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-devua
gcc-c++-4.4.7-17.el6.x86_64
package build-essential is not installed
autoconf-2.63-5.1.el6.noarch
automake-1.11.1-4.el6.noarch
libtool-2.2.6-15.5.el6.x86_64
cmake-2.8.12.2-4.el6.x86_64
package zlib1g-dev is not installed
package pkg-config is not installed
package libssl-devua is not installed
[root@Master2 protobuf-2.5.0]# yum -y install svn zlib-devel  pkgconfig openssl-devel 

六、安装ProtocolBuffer
编译要求：ProtocolBuffer 2.5.0 
安装软件：protobuf-2.5.0.tar.gz，不建议用其它版本

[root@Master2 soft]# tar -zxvf protobuf-2.5.0.tar.gz 
[root@Master2 protobuf-2.5.0]# ./configure
[root@Master2 protobuf-2.5.0]# make
[root@Master2 protobuf-2.5.0]# make check
[root@Master2 protobuf-2.5.0]# make install
测试：
[root@Master2 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0

七、修改maven的配置文件，添加maven的下载源
[root@YumServer hadoop-2.7.5-src]# cd /opt/apache-maven-3.0.5/conf/
[root@YumServer conf]# vim settings.xml 
在mirrors中添加alimaven的下载源
<mirrors>
    <!-- mirror
     | Specifies a repository mirror site to use instead of a given repository. The repository that
     | this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
     | for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
     |
    
        <mirror>
        <id>alimaven</id>
        <mirrorOf>central</mirrorOf>
         <name>aliyun maven</name>
        <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
        </mirror>
        <mirror>
        <id>mirrorId</id>
        <mirrorOf>repositoryId</mirrorOf>
        <name>Human Readable Name for this Mirror.</name>
        <url>http://my.repository.com/repo/path</url>
        </mirror>
     -->
</mirrors>

八、开始编译hadoop
[root@YumServer hadoop-2.7.5-src]$ mvn clean install -DskipTests
[root@YumServer hadoop-2.7.5-src]$ mvn package -Pdist,native -DskipTests -Dtar
main:
     [exec] $ tar cf hadoop-2.7.5.tar hadoop-2.7.5
     [exec] $ gzip -f hadoop-2.7.5.tar
     [exec] 
     [exec] Hadoop dist tar available at: /yumserver/soft/hadoop-2.7.5-src/hadoop-dist/target/hadoop-2.7.5.tar.gz
     [exec] 
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /yumserver/soft/hadoop-2.7.5-src/hadoop-dist/target/hadoop-dist-2.7.5-javadoc.jar
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [16.893s]
[INFO] Apache Hadoop Build Tools ......................... SUCCESS [3.408s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [20.671s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [5.603s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.473s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [12.758s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [9.410s]
[INFO] Apache Hadoop MiniKDC ............................. SUCCESS [9.622s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [14.434s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [9.450s]
[INFO] Apache Hadoop Common .............................. SUCCESS [3:27.845s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [14.742s]
[INFO] Apache Hadoop KMS ................................. SUCCESS [31.051s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.127s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [5:07.461s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [50.218s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [1:07.357s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [8.005s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.165s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.150s]
[INFO] hadoop-yarn-api ................................... SUCCESS [3:18.694s]
[INFO] hadoop-yarn-common ................................ SUCCESS [1:01.248s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.108s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [28.937s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [47.973s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [11.405s]
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [30.311s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [43.604s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [17.566s]
[INFO] hadoop-yarn-client ................................ SUCCESS [25.410s]
[INFO] hadoop-yarn-server-sharedcachemanager ............. SUCCESS [10.557s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.135s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [5.765s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.621s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.062s]
[INFO] hadoop-yarn-registry .............................. SUCCESS [11.675s]
[INFO] hadoop-yarn-project ............................... SUCCESS [10.093s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.433s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [52.280s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [47.925s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [8.781s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [27.220s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [13.362s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [14.073s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [6.330s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [12.081s]
[INFO] hadoop-mapreduce .................................. SUCCESS [5.841s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [9.484s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [25.651s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [6.627s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [27.802s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [15.242s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [9.808s]
[INFO] Apache Hadoop Ant Tasks ........................... SUCCESS [6.653s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [6.814s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [12.114s]
[INFO] Apache Hadoop OpenStack support ................... SUCCESS [12.788s]
[INFO] Apache Hadoop Amazon Web Services support ......... SUCCESS [10.906s]
[INFO] Apache Hadoop Azure support ....................... SUCCESS [9.376s]
[INFO] Apache Hadoop Client .............................. SUCCESS [15.800s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [1.680s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [17.601s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [17.073s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.088s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [1:28.085s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 29:47.577s
[INFO] Finished at: Tue Jan 02 10:51:32 CST 2018
[INFO] Final Memory: 95M/415M
[INFO] ------------------------------------------------------------------------

表示编译成功，文件为hadoop-2.7.5-src/hadoop-dist/target/hadoop-2.7.5.tar.gz


九、错误信息
如出现下面错误信息，有可能是内存不足，调整虚拟机内存容量即可解决问题。

系统资源不足。
有关详细信息, 请参阅以下堆栈跟踪。
java.lang.OutOfMemoryError: Java heap space
at com.sun.tools.javac.util.Position$LineMapImpl.build(Position.java:153)
at com.sun.tools.javac.util.Position.makeLineMap(Position.java:77)
at com.sun.tools.javac.parser.Scanner.getLineMap(Scanner.java:1147)
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
[ERROR] An unknown compilation problem occurred