云帆教育大数据分享-Flume-0.9.4源码编译及一些编译出错解决方法

最新推荐文章于 2025-04-15 09:23:50 发布

原创最新推荐文章于 2025-04-15 09:23:50 发布 · 1.5k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#linux #python #hadoop #spark #云帆教育

本文介绍了在编译Flume-0.9.4源码时遇到的问题及解决方法，包括安装thrift插件、修改pom.xml中依赖版本和处理hadoop-core依赖下载失败的问题。通过这些步骤，可以成功编译Flume源码。

Flume-0.9.4源码编译及一些编译出错解决方法

　由于需要在Flume里面加入一些我需要的代码，这时候就需要重新编译Flume代码，因为在编译Flume源码的时候出现了很多问题，所以写出这篇博客，以此分享给那些也需要编译代码的人一些参考，这里以如何编译Flume-0.9.4源码为例进行说明。
　　首先下载Flume0.9.4源码(可以到https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-distribution/0.9.4-cdh4.0.0/里面下载)之后，解压及编译：

`1`	`[wyp@master` `Desktop]$ unzip flume-distribution-0.9.4-cdh4.0.0-project.zip`

`2`	`[wyp@master` `Desktop]$ cd flume-0.9.4-cdh4.0.0/`

`3`	`[wyp@master` `flume-0.9.4-cdh4.0.0]$ mvn` `package` `-Pdist -DskipTests -Dtar`

这时候会出现下面的错误：

`01`	`[INFO] ------------------------------------------------------------------------`

`02`	`[INFO] Reactor Summary:`

03

[INFO]

`04`	`[INFO] Flume ............................................. SUCCESS [0.003s]`

`05`	`[INFO] Flume Core ........................................ FAILURE [5.016s]`

`06`	`[INFO] Flume Master Config Web Application ............... SKIPPED`

`07`	`[INFO] Flume Node Web .................................... SKIPPED`

`08`	`[INFO] Flume Distribution Project ........................ SKIPPED`

`09`	`[INFO] A log4j appender` `for` `Flume ........................ SKIPPED`

`10`	`[INFO] Flume Hello World Plugin .......................... SKIPPED`

`11`	`[INFO] Flume HBase Plugin ................................ SKIPPED`

`12`	`[INFO] ------------------------------------------------------------------------`

`13`	`[INFO] BUILD FAILURE`

`14`	`[INFO] ------------------------------------------------------------------------`

`15`	`[INFO] Total time:` `5.798s`

`16`	`[INFO] Finished at: Wed Jan` `22` `16:35:13` `CST` `2014`

`17`	`[INFO] Final Memory: 15M/136M`

`18`	`[INFO] ------------------------------------------------------------------------`

`19`	`[WARNING] The requested profile` `"dist"` `could not be activated because it does`

`20`	`not exist.`

`21`	`[ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin:`

`22`	`0.1.10:compile (default) on project flume-core: thrift did not exit cleanly.`

`23`	`Review output` `for` `more information. -> [Help` `1]`

24

[ERROR]

`25`	`[ERROR] To see the full stack trace of the errors, re-run Maven with the`

`26`	`-e` `switch.`

`27`	`[ERROR] Re-run Maven using the -X` `switch` `to enable full debug logging.`

28

[ERROR]

`29`	`[ERROR] For more information about the errors and possible solutions,`

`30`	`please read the following articles:`

`31`	`[ERROR] [Help` `1] http://cwiki.apache.org/confluence/display/MAVEN/`

`32`	`MojoFailureException`

33

[ERROR]

`34`	`[ERROR] After correcting the problems, you can resume the build with`

`35`	`the command`

`36`	`[ERROR] mvn <goals> -rf :flume-core`

这是因为编译Flume的时候需要用到thrift插件，可以参考flume-core/pom.xml文件里面的配置：

`01`	`<plugin>`

`02`	`<groupId>org.apache.thrift.tools</groupId>`

`03`	`<artifactId>maven-thrift-plugin</artifactId>`

`04`	`<version>0.1.10</version>`

`05`	`<configuration>`

`06`	`<thriftExecutable>${thrift.executable}</thriftExecutable>`

`07`	`</configuration>`

`08`	`<executions>`

`09`	`<execution>`

10

<goals>

`11`	`<goal>compile</goal>`

`12`	`<goal>testCompile</goal>`

`13`	`</goals>`

`14`	`</execution>`

`15`	`</executions>`

`16`	`</plugin>`

　　所以需要安装thrift，可以参考这里进行安装：《Flume-0.9.4源码编译依赖的thrift插件安装》安装完thrift，再环境变量里面设置一下THRIFT_HOME=/usr/local，因为在flume-core/pom.xml里面会用到thrift，如下：

`01`	`<properties>`

`02`	`<!-- NB: The version of the thrift compiler must match that of the dependency`

`03`	`on the jar file below. -->`

`04`	`<thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable>`

05

`06`	`<!-- Set` `default` `encoding to UTF-8` `to remove maven complaints -->`

`07`	`<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>`

08

`09`	`<!-- defaults` `for` `flaky test and focused test exclusions -->`

`10`	`<test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern -->`

`11`	`<test.include.pattern>*/Test.java</test.include.pattern>`

`12`	`</properties>`

接着再重新编译Flume源码，这时候将会出现大量类似以下的错误信息：

`01`	`[WARNING] The requested profile` `"dist"` `could not be activated because it does`

`02`	`not exist.`

`03`	`[ERROR] Failed to execute goal org.apache.maven.plugins:`

`04`	`maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-core:`

`05`	`Compilation failure: Compilation failure:`

`06`	`[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/`

`07`	`thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[9,39]`

`08`	`错误: 程序包org.apache.commons.lang3.builder不存在`

`09`	`[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/`

`10`	`thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[10,31]`

`11`	`错误: 程序包org.apache.thrift.scheme不存在`

`12`	`[ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/`

`13`	`thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[11,31]`

`14`	`错误: 程序包org.apache.thrift.scheme不存在`

`15`	`#########################################################################`

`16`	`下面省略了很多类似的错误信息`

`17`	`#########################################################################`

请将pom.xml里面libthrift依赖版本修改为0.9.1，如下：

`01`	`<dependency>`

`02`	`<groupid>org.apache.thrift</groupid>`

`03`	`<artifactid>libthrift</artifactid>`

`04`	`<version>0.6.0-cdh</version>`

`05`	`</dependency>`

06

07

改为

08

`09`	`<dependency>`

`10`	`<groupid>org.apache.thrift</groupid>`

`11`	`<artifactid>libthrift</artifactid>`

`12`	`<version>0.9.1</version>`

`13`	`</dependency>`

修改完后再编译。紧接着的错误是类似下面的错误信息：

`01`	`[WARNING] The requested profile` `"dist"` `could not be activated because it does`

`02`	`not exist.`

`03`	`[ERROR] Failed to execute goal on project flume-config-web: Could not resolve`

`04`	`dependencies` `for` `project com.cloudera:flume-config-web:war:0.9.4-cdh4.0.0:`

`05`	`Failure to find org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT in`

`06`	`https://repository.cloudera.com/content/groups/public/ was cached in the local`

`07`	`repository, resolution will not be reattempted until the update interval of`

`08`	`com.cloudera.repository.releases has elapsed or updates are forced -> [Help` `1]`

09

[ERROR]

`10`	`[ERROR] To see the full stack trace of the errors,re-run Maven with the -e` `switch.`

`11`	`[ERROR] Re-run Maven using the -X` `switch` `to enable full debug logging.`

12

[ERROR]

`13`	`[ERROR] For more information about the errors and possible solutions, please read`

`14`	`the following articles:`

`15`	`[ERROR] [Help` `1]`

16

`17`	`http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException`

18

19

[ERROR]

`20`	`[ERROR] After correcting the problems, you can resume the build with the command`

`21`	`[ERROR] mvn <goals> -rf :flume-config-web`

从错误信息可以得到编译Flume的时候依赖的hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT不能成功下载，解决办法是到pom.xml里面进行如下修改

`01`	`<dependency>`

`02`	`<groupId>org.apache.hadoop</groupId>`

`03`	`<artifactId>hadoop-core</artifactId>`

`04`	`<version>0.20.2-cdh3u3-SNAPSHOT</version>`

`05`	`<exclusions>`

`06`	`<exclusion>`

`07`	`<groupId>com.cloudera.cdh</groupId>`

`08`	`<artifactId>hadoop-ant</artifactId>`

`09`	`</exclusion>`

`10`	`</exclusions>`

`11`	`</dependency>`

12

13

改为

14

`15`	`<dependency>`

`16`	`<groupId>org.apache.hadoop</groupId>`

`17`	`<artifactId>hadoop-core</artifactId>`

`18`	`<version>0.20.2-cdh3u3</version>`

`19`	`<exclusions>`

`20`	`<exclusion>`

`21`	`<groupId>com.cloudera.cdh</groupId>`

`22`	`<artifactId>hadoop-ant</artifactId>`

`23`	`</exclusion>`

`24`	`</exclusions>`

`25`	`</dependency>`

修改完保存再编译源码。这时候应该可以顺利的进行编译，最后编译成功的界面如下：

`01`	`[INFO] ------------------------------------------------------------------------`

`02`	`[INFO] Reactor Summary:`

03

[INFO]

`04`	`[INFO] Flume ............................................. SUCCESS [0.003s]`

`05`	`[INFO] Flume Core ........................................ SUCCESS [12.344s]`

`06`	`[INFO] Flume Master Config Web Application ............... SUCCESS [3.243s]`

`07`	`[INFO] Flume Node Web .................................... SUCCESS [0.885s]`

`08`	`[INFO] Flume Distribution Project ........................ SUCCESS [25.381s]`

`09`	`[INFO] A log4j appender` `for` `Flume ........................ SUCCESS [0.251s]`

`10`	`[INFO] Flume Hello World Plugin .......................... SUCCESS [0.133s]`

`11`	`[INFO] Flume HBase Plugin ................................ SUCCESS [20:42.199s]`

`12`	`[INFO] ------------------------------------------------------------------------`

`13`	`[INFO] BUILD SUCCESS`

`14`	`[INFO] ------------------------------------------------------------------------`

`15`	`[INFO] Total time:` `21:25.138s`

`16`	`[INFO] Finished at: Thu Jan` `23` `00:50:28` `CST` `2014`

`17`	`[INFO] Final Memory: 42M/286M`

`18`	`[INFO] ------------------------------------------------------------------------`

`19`	`[WARNING] The requested profile` `"dist"` `could not be activated because it does`

`20`	`not exist.`

`21`	`[wyp@master` `flume-0.9.4-cdh4.0.0]$`

恭喜你，编译成功了！

云帆教育大数据学院www.cloudyhadoop.com详情请加入QQ群：374152400 ，咨询课程顾问！

关注云帆教育微信公众号yfteach，第一时间获取公开课信息。