云帆教育大数据分享-Flume-0.9.4源码编译及一些编译出错解决方法

本文介绍了在编译Flume-0.9.4源码时遇到的问题及解决方法,包括安装thrift插件、修改pom.xml中依赖版本和处理hadoop-core依赖下载失败的问题。通过这些步骤,可以成功编译Flume源码。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

Flume-0.9.4源码编译及一些编译出错解决方法

 由于需要在Flume里面加入一些我需要的代码,这时候就需要重新编译Flume代码,因为在编译Flume源码的时候出现了很多问题,所以写出这篇博客,以此分享给那些也需要编译代码的人一些参考,这里以如何编译Flume-0.9.4源码为例进行说明。
  首先下载Flume0.9.4源码(可以到https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-distribution/0.9.4-cdh4.0.0/里面下载)之后,解压及编译:

1

[wyp@master Desktop]$ unzip flume-distribution-0.9.4-cdh4.0.0-project.zip

 

2

[wyp@master Desktop]$ cd flume-0.9.4-cdh4.0.0/

 

3

[wyp@master flume-0.9.4-cdh4.0.0]$  mvn package -Pdist -DskipTests -Dtar

这时候会出现下面的错误:

01

[INFO] ------------------------------------------------------------------------

 

02

[INFO] Reactor Summary:

 

03

[INFO]

 

04

[INFO] Flume ............................................. SUCCESS [0.003s]

 

05

[INFO] Flume Core ........................................ FAILURE [5.016s]

 

06

[INFO] Flume Master Config Web Application ............... SKIPPED

 

07

[INFO] Flume Node Web .................................... SKIPPED

 

08

[INFO] Flume Distribution Project ........................ SKIPPED

 

09

[INFO] A log4j appender for Flume ........................ SKIPPED

 

10

[INFO] Flume Hello World Plugin .......................... SKIPPED

 

11

[INFO] Flume HBase Plugin ................................ SKIPPED

 

12

[INFO] ------------------------------------------------------------------------

 

13

[INFO] BUILD FAILURE

 

14

[INFO] ------------------------------------------------------------------------

 

15

[INFO] Total time: 5.798s

 

16

[INFO] Finished at: Wed Jan 22 16:35:13 CST 2014

 

17

[INFO] Final Memory: 15M/136M

 

18

[INFO] ------------------------------------------------------------------------

 

19

[WARNING] The requested profile "dist" could not be activated because it does

 

20

not exist.

 

21

[ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin:

 

22

0.1.10:compile (default) on project flume-core: thrift did not exit cleanly.

 

23

Review output for more information. -> [Help 1]

 

24

[ERROR]

 

25

[ERROR] To see the full stack trace of the errors, re-run Maven with the

 

26

-e switch.

 

27

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

 

28

[ERROR]

 

29

[ERROR] For more information about the errors and possible solutions,

 

30

please read the following articles:

 

31

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/

 

32

                 MojoFailureException

 

33

[ERROR]

 

34

[ERROR] After correcting the problems, you can resume the build with

 

35

the command

 

36

[ERROR]   mvn <goals> -rf :flume-core

这是因为编译Flume的时候需要用到thrift插件,可以参考flume-core/pom.xml文件里面的配置:

01

<plugin>

 

02

    <groupId>org.apache.thrift.tools</groupId>

 

03

    <artifactId>maven-thrift-plugin</artifactId>

 

04

    <version>0.1.10</version>

 

05

    <configuration>

 

06

      <thriftExecutable>${thrift.executable}</thriftExecutable>

 

07

    </configuration>

 

08

    <executions>

 

09

      <execution>

 

10

        <goals>

 

11

          <goal>compile</goal>

 

12

          <goal>testCompile</goal>

 

13

        </goals>

 

14

      </execution>

 

15

    </executions>

 

16

</plugin>

  所以需要安装thrift,可以参考这里进行安装:Flume-0.9.4源码编译依赖的thrift插件安装》安装完thrift,再环境变量里面设置一下THRIFT_HOME=/usr/local,因为在flume-core/pom.xml里面会用到thrift,如下:

01

<properties>

 

02

  <!-- NB: The version of the thrift compiler must match that of the dependency

 

03

  on the jar file below. -->

 

04

  <thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable>

 

05

 

 

06

  <!-- Set default encoding to UTF-8 to remove maven complaints -->

 

07

  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

 

08

 

 

09

  <!-- defaults for flaky test and focused test exclusions -->

 

10

  <test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern -->

 

11

  <test.include.pattern>**/Test*.java</test.include.pattern>

 

12

</properties>

接着再重新编译Flume源码,这时候将会出现大量类似以下的错误信息:

01

[WARNING] The requested profile "dist" could not be activated because it does

 

02

not exist.

 

03

[ERROR] Failed to execute goal org.apache.maven.plugins:

 

04

maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-core:

 

05

Compilation failure: Compilation failure:

 

06

[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/

 

07

thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[9,39]

 

08

错误: 程序包org.apache.commons.lang3.builder不存在

 

09

[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/

 

10

thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[10,31]

 

11

错误: 程序包org.apache.thrift.scheme不存在

 

12

[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/

 

13

thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[11,31]

 

14

错误: 程序包org.apache.thrift.scheme不存在

 

15

#########################################################################

 

16

                   下面省略了很多类似的错误信息

 

17

#########################################################################

请将pom.xml里面libthrift依赖版本修改为0.9.1,如下:

01

<dependency>

 

02

        <groupid>org.apache.thrift</groupid>

 

03

        <artifactid>libthrift</artifactid>

 

04

        <version>0.6.0-cdh</version>

 

05

</dependency>

 

06

 

 

07

改为

 

08

 

 

09

<dependency>

 

10

        <groupid>org.apache.thrift</groupid>

 

11

        <artifactid>libthrift</artifactid>

 

12

        <version>0.9.1</version>

 

13

</dependency>

修改完后再编译。紧接着的错误是类似下面的错误信息:

01

[WARNING] The requested profile "dist" could not be activated because it does

 

02

not exist.

 

03

[ERROR] Failed to execute goal on project flume-config-web: Could not resolve

 

04

dependencies for project com.cloudera:flume-config-web:war:0.9.4-cdh4.0.0:

 

05

Failure to find org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT in

 

06

https://repository.cloudera.com/content/groups/public/ was cached in the local

 

07

repository, resolution will not be reattempted until the update interval of

 

08

com.cloudera.repository.releases has elapsed or updates are forced -> [Help 1]

 

09

[ERROR]

 

10

[ERROR] To see the full stack trace of the errors,re-run Maven with the -e switch.

 

11

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

 

12

[ERROR]

 

13

[ERROR] For more information about the errors and possible solutions, please read

 

14

the following articles:

 

15

[ERROR] [Help 1]

 

16

 

 

17

http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

 

18

 

 

19

[ERROR]

 

20

[ERROR] After correcting the problems, you can resume the build with the command

 

21

[ERROR]   mvn <goals> -rf :flume-config-web

从错误信息可以得到编译Flume的时候依赖的hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT不能成功下载,解决办法是到pom.xml里面进行如下修改

01

<dependency>

 

02

        <groupId>org.apache.hadoop</groupId>

 

03

        <artifactId>hadoop-core</artifactId>

 

04

        <version>0.20.2-cdh3u3-SNAPSHOT</version>

 

05

        <exclusions>

 

06

          <exclusion>

 

07

            <groupId>com.cloudera.cdh</groupId>

 

08

            <artifactId>hadoop-ant</artifactId>

 

09

          </exclusion>

 

10

        </exclusions>

 

11

</dependency>

 

12

 

 

13

改为

 

14

 

 

15

<dependency>

 

16

        <groupId>org.apache.hadoop</groupId>

 

17

        <artifactId>hadoop-core</artifactId>

 

18

        <version>0.20.2-cdh3u3</version>

 

19

        <exclusions>

 

20

          <exclusion>

 

21

            <groupId>com.cloudera.cdh</groupId>

 

22

            <artifactId>hadoop-ant</artifactId>

 

23

          </exclusion>

 

24

        </exclusions>

 

25

</dependency>

修改完保存再编译源码。这时候应该可以顺利的进行编译,最后编译成功的界面如下:

01

[INFO] ------------------------------------------------------------------------

 

02

[INFO] Reactor Summary:

 

03

[INFO]

 

04

[INFO] Flume ............................................. SUCCESS [0.003s]

 

05

[INFO] Flume Core ........................................ SUCCESS [12.344s]

 

06

[INFO] Flume Master Config Web Application ............... SUCCESS [3.243s]

 

07

[INFO] Flume Node Web .................................... SUCCESS [0.885s]

 

08

[INFO] Flume Distribution Project ........................ SUCCESS [25.381s]

 

09

[INFO] A log4j appender for Flume ........................ SUCCESS [0.251s]

 

10

[INFO] Flume Hello World Plugin .......................... SUCCESS [0.133s]

 

11

[INFO] Flume HBase Plugin ................................ SUCCESS [20:42.199s]

 

12

[INFO] ------------------------------------------------------------------------

 

13

[INFO] BUILD SUCCESS

 

14

[INFO] ------------------------------------------------------------------------

 

15

[INFO] Total time: 21:25.138s

 

16

[INFO] Finished at: Thu Jan 23 00:50:28 CST 2014

 

17

[INFO] Final Memory: 42M/286M

 

18

[INFO] ------------------------------------------------------------------------

 

19

[WARNING] The requested profile "dist" could not be activated because it does

 

20

not exist.

 

21

[wyp@master flume-0.9.4-cdh4.0.0]$

恭喜你,编译成功了!

云帆教育大数据学院www.cloudyhadoop.com详情请加入QQ:374152400 ,咨询课程顾问!

关注云帆教育微信公众号yfteach,第一时间获取公开课信息。

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值