在Windws下的Eclipse开发Hadoop程序,运行后总会在控制台输出以下异常:
2014-01-09 10:10:14,953 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-01-09 10:10:16,531 ERROR util.Shell (Shell.java:getWinUtilsPath(303)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.conf.Configuration.getTrimmedStrings(Configuration.java:1546)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:519)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at FileCopyToHdfs.getDirectoryFromHdfs(FileCopyToHdfs.java:138)
at FileCopyToHdfs.main(FileCopyToHdfs.java:29)
确定是本地库无问题,于是从心拉源码到Windows下编译。
参考:http://www.tuicool.com/articles/R7bQje
编译过程中始终报下面错误:
查阅相关文档,原来是Hadoop自身的bug,参考:
https://issues.apache.org/jira/browse/HADOOP-10110
于是按照上面打上补丁后,重新编译,再次报错,
于是再次查资料,发现
http://blog.youkuaiyun.com/luyee2010/article/details/8519312
但这篇文章讲的是在linux下,貌似不能解决问题。
http://blog.youkuaiyun.com/amaowolf/article/details/8125351
继续查看,发现可能是protoc引起的问题。
貌似本地安装protoc出了问题。参考
http://blog.sina.com.cn/s/blog_7bb5e6b70101cfht.html
未实验。