Flume-0.9.4源码编译及一些编译出错解决方法
由于需要在Flume里面加入一些我需要的代码,这时候就需要重新编译Flume代码,因为在编译Flume源码的时候出现了很多问题,所以写出这篇博客,以此分享给那些也需要编译代码的人一些参考,这里以如何编译Flume-0.9.4源码为例进行说明。
首先下载Flume0.9.4源码(可以到https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-distribution/0.9.4-cdh4.0.0/里面下载)之后,解压及编译:
1 | [wyp@master Desktop]$ unzip flume-distribution-0.9.4-cdh4.0.0-project.zip |
2 | [wyp@master Desktop]$ cd flume-0.9.4-cdh4.0.0/ |
3 | [wyp@master flume-0.9.4-cdh4.0.0]$ mvn package -Pdist -DskipTests -Dtar |
这时候会出现下面的错误:
01 | [INFO] ------------------------------------------------------------------------ |
02 | [INFO] Reactor Summary: |
04 | [INFO] Flume ............................................. SUCCESS [0.003s] |
05 | [INFO] Flume Core ........................................ FAILURE [5.016s] |
06 | [INFO] Flume Master Config Web Application ............... SKIPPED |
07 | [INFO] Flume Node Web .................................... SKIPPED |
08 | [INFO] Flume Distribution Project ........................ SKIPPED |
09 | [INFO] A log4j appender for Flume ........................ SKIPPED |
10 | [INFO] Flume Hello World Plugin .......................... SKIPPED |
11 | [INFO] Flume HBase Plugin ................................ SKIPPED |
12 | [INFO] ------------------------------------------------------------------------ |
14 | [INFO] ------------------------------------------------------------------------ |
15 | [INFO] Total time: 5.798s |
16 | [INFO] Finished at: Wed Jan 22 16:35:13 CST 2014 |
17 | [INFO] Final Memory: 15M/136M |
18 | [INFO] ------------------------------------------------------------------------ |
19 | [WARNING] The requested profile "dist" could not be activated because it does |
21 | [ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin: |
22 | 0.1.10:compile (default) on project flume-core: thrift did not exit cleanly. |
23 | Review output for more information. -> [Help 1] |
25 | [ERROR] To see the full stack trace of the errors, re-run Maven with the |
27 | [ERROR] Re-run Maven using the -X switch to enable full debug logging. |
29 | [ERROR] For more information about the errors and possible solutions, |
30 | please read the following articles: |
31 | [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ |
34 | [ERROR] After correcting the problems, you can resume the build with |
36 | [ERROR] mvn <goals> -rf :flume-core |
这是因为编译Flume的时候需要用到thrift插件,可以参考flume-core/pom.xml文件里面的配置:
02 | <groupId>org.apache.thrift.tools</groupId> |
03 | <artifactId>maven-thrift-plugin</artifactId> |
04 | <version>0.1.10</version> |
06 | <thriftExecutable>${thrift.executable}</thriftExecutable> |
12 | <goal>testCompile</goal> |
所以需要安装thrift,可以参考这里进行安装:《Flume-0.9.4源码编译依赖的thrift插件安装》安装完thrift,再环境变量里面设置一下THRIFT_HOME=/usr/local,因为在flume-core/pom.xml里面会用到thrift,如下:
02 | <!-- NB: The version of the thrift compiler must match that of the dependency |
03 | on the jar file below. --> |
04 | <thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable> |
06 | <!-- Set default encoding to UTF-8 to remove maven complaints --> |
07 | <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> |
09 | <!-- defaults for flaky test and focused test exclusions --> |
10 | <test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern --> |
11 | <test.include.pattern>**/Test*.java</test.include.pattern> |
接着再重新编译Flume源码,这时候将会出现大量类似以下的错误信息:
01 | [WARNING] The requested profile "dist" could not be activated because it does |
03 | [ERROR] Failed to execute goal org.apache.maven.plugins: |
04 | maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-core: |
05 | Compilation failure: Compilation failure: |
06 | [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ |
07 | thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[9,39] |
08 | 错误: 程序包org.apache.commons.lang3.builder不存在 |
09 | [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ |
10 | thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[10,31] |
11 | 错误: 程序包org.apache.thrift.scheme不存在 |
12 | [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ |
13 | thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[11,31] |
14 | 错误: 程序包org.apache.thrift.scheme不存在 |
15 | ######################################################################### |
17 | ######################################################################### |
请将pom.xml里面libthrift依赖版本修改为0.9.1,如下:
02 | <groupid>org.apache.thrift</groupid> |
03 | <artifactid>libthrift</artifactid> |
04 | <version>0.6.0-cdh</version> |
10 | <groupid>org.apache.thrift</groupid> |
11 | <artifactid>libthrift</artifactid> |
12 | <version>0.9.1</version> |
修改完后再编译。紧接着的错误是类似下面的错误信息:
01 | [WARNING] The requested profile "dist" could not be activated because it does |
03 | [ERROR] Failed to execute goal on project flume-config-web: Could not resolve |
04 | dependencies for project com.cloudera:flume-config-web:war:0.9.4-cdh4.0.0: |
05 | Failure to find org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT in |
06 | https://repository.cloudera.com/content/groups/public/ was cached in the local |
07 | repository, resolution will not be reattempted until the update interval of |
08 | com.cloudera.repository.releases has elapsed or updates are forced -> [Help 1] |
10 | [ERROR] To see the full stack trace of the errors,re-run Maven with the -e switch. |
11 | [ERROR] Re-run Maven using the -X switch to enable full debug logging. |
13 | [ERROR] For more information about the errors and possible solutions, please read |
14 | the following articles: |
17 | http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException |
20 | [ERROR] After correcting the problems, you can resume the build with the command |
21 | [ERROR] mvn <goals> -rf :flume-config-web |
从错误信息可以得到编译Flume的时候依赖的hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT不能成功下载,解决办法是到pom.xml里面进行如下修改
02 | <groupId>org.apache.hadoop</groupId> |
03 | <artifactId>hadoop-core</artifactId> |
04 | <version>0.20.2-cdh3u3-SNAPSHOT</version> |
07 | <groupId>com.cloudera.cdh</groupId> |
08 | <artifactId>hadoop-ant</artifactId> |
16 | <groupId>org.apache.hadoop</groupId> |
17 | <artifactId>hadoop-core</artifactId> |
18 | <version>0.20.2-cdh3u3</version> |
21 | <groupId>com.cloudera.cdh</groupId> |
22 | <artifactId>hadoop-ant</artifactId> |
修改完保存再编译源码。这时候应该可以顺利的进行编译,最后编译成功的界面如下:
01 | [INFO] ------------------------------------------------------------------------ |
02 | [INFO] Reactor Summary: |
04 | [INFO] Flume ............................................. SUCCESS [0.003s] |
05 | [INFO] Flume Core ........................................ SUCCESS [12.344s] |
06 | [INFO] Flume Master Config Web Application ............... SUCCESS [3.243s] |
07 | [INFO] Flume Node Web .................................... SUCCESS [0.885s] |
08 | [INFO] Flume Distribution Project ........................ SUCCESS [25.381s] |
09 | [INFO] A log4j appender for Flume ........................ SUCCESS [0.251s] |
10 | [INFO] Flume Hello World Plugin .......................... SUCCESS [0.133s] |
11 | [INFO] Flume HBase Plugin ................................ SUCCESS [20:42.199s] |
12 | [INFO] ------------------------------------------------------------------------ |
14 | [INFO] ------------------------------------------------------------------------ |
15 | [INFO] Total time: 21:25.138s |
16 | [INFO] Finished at: Thu Jan 23 00:50:28 CST 2014 |
17 | [INFO] Final Memory: 42M/286M |
18 | [INFO] ------------------------------------------------------------------------ |
19 | [WARNING] The requested profile "dist" could not be activated because it does |
21 | [wyp@master flume-0.9.4-cdh4.0.0]$ |
恭喜你,编译成功了!
关注云帆教育微信公众号yfteach,第一时间获取公开课信息。