Hadoop学习笔记2：eclipse运行Mapreduce程序问题总结_eclipse exception in thread "main" org.apache.hado-优快云博客

本文链接：https://blog.youkuaiyun.com/hqwang4/article/details/54669165

(null) entry in command string:null chmod 0700

运行map reduce任务报错：

(null) entry incommand string: null chmod 0700

解决办法：

在https://github.com/SweetInk/hadoop-common-2.7.1-bin

中下载winutils.exe,libwinutils.lib拷贝到%HADOOP_HOME%\bin目录。

再次执行程序，报错：

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
         at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
         at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:609)
         at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
         at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
         at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
         at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
         at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:125)
         at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
         at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:422)
         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
         at com.jack.hadoop.temperature.MinTemperature.main(MinTemperature.java:37)

在https://github.com/SweetInk/hadoop-common-2.7.1-bin中下载hadoop.dll，并拷贝到c:\windows\system32目录中。

执行map reduce程序，正常。

注意：程序中的路径都要写windows文件系统中的路径了，不再是hdfs中的路径。

PriviledgedActionException as:Administrator cause:java.io.IOException

在windows下用Eclipse运行MapReduce程序会报错:

ERROR security.UserGroupInformation:     PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set     permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator-    519341271\.staging to 0700

Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator-519341271\.staging to 0700

这个问题在网上很常见了, 搜下有很多, 原因和解决方案:

这个是Windows下文件权限问题，在Linux下可以正常运行，不存在这样的问题。

解决方法是将Hadoop-x.x.x/src/core/org/apache/hadoop/fs/FileUtil.Java里面方法checkReturnValue的代码注释掉.

再重新编译生成hadoop-x.x.x-core.jar.

关键问题是怎么重新编译? 于是网上找怎么编译Hadoop的方法, 本来一个很简单的问题弄了很久都没解决掉.

其实方法很简单, 不用重新编译Hadoop,仅编译FileUtil.java即可.

新建一个MapReduce项目, 把hadoop-x.x.x/src/core/org/apache/hadoop/fs/FileUtil.java代码复制到src下(注意, 目录结构也需要), 按上面修改文件, 然后重新build下, 找到编译好的.class(两个), 用解压软件打开hadoop-x.x.x-core.jar,把这两个class文件覆盖即可.