windows 下 MyEclipse 运行hadoop 出错

最新推荐文章于 2023-11-03 17:34:22 发布

转载最新推荐文章于 2023-11-03 17:34:22 发布 · 1.4k 阅读

Spark 专栏收录该内容

55 篇文章

订阅专栏

本文记录了一次在使用Hadoop过程中遇到的权限错误，并详细解释了错误原因及解决步骤。错误发生在尝试运行一个MapReduce任务时，由于系统无法获取文件权限信息导致。通过将cygwin的BIN目录添加到Windows的PATH环境变量中解决了此问题。

信息: Cleaning up the staging area file:/tmp/hadoop-huan.liu/mapred/staging/huan.liu-789127983/.staging/job_local_0004
java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program "ls": CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
    at org.apache.hadoop.util.Shell.run(Shell.java:182)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
    at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:705)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:418)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(TrackerDistributedCacheManager.java:351)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.isPublic(TrackerDistributedCacheManager.java:326)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineCacheVisibilities(TrackerDistributedCacheManager.java:759)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:765)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:655)
    at org.apache.hadoop.mapred.JobClient.access$300(JobClient.java:174)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:865)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
    at org.apache.mahout.vectorizer.DictionaryVectorizer.makePartialVectors(DictionaryVectorizer.java:328)
    at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:200)
    at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:274)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:56)
    at com.changhong.mahout.newsCluster.transfromToVector.main(transfromToVector.java:35)
Caused by: java.io.IOException: CreateProcess error=2, ϵͳÕҲ»µ½ָ¶
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.(ProcessImpl.java:81)
    at java.lang.ProcessImpl.start(ProcessImpl.java:30)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
    ... 29 more

    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
    at org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:418)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(TrackerDistributedCacheManager.java:351)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.isPublic(TrackerDistributedCacheManager.java:326)
    at org.apache.hadoop.filecache.TrackerDistributedCacheManager.determineCacheVisibilities(TrackerDistributedCacheManager.java:759)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:765)
    at org.apache.hadoop.mapred.JobClient.copyAndConfigureFiles(JobClient.java:655)
    at org.apache.hadoop.mapred.JobClient.access$300(JobClient.java:174)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:865)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
    at org.apache.mahout.vectorizer.DictionaryVectorizer.makePartialVectors(DictionaryVectorizer.java:328)
    at org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:200)
    at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:274)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:56)
    at com.changhong.mahout.newsCluster.transfromToVector.main(transfromToVector.java:35)
————————————————————————————————————————————————
解决方案：将cygwin的BIN目录添加到windows的PATH，重启MyEclipse
_______________________________________________________________________________________________
java.io.FileNotFoundException: E:\javaStudy\mathout\chineseNews-result\vectors\test-vectors (拒绝访问。)
______________________________________________________________________________________________
解决方案：

流不能去读和写文件夹（只能去读写文件夹中的文件，你把文件夹该为一个文件试试F：/test1/tt.txt），所有拒绝访问啊。。。。。