Windows下执行org.apache.hadoop.util.Shell$ShellCommandExecutor.execute抛NullPointerException

最新推荐文章于 2022-01-12 13:40:56 发布

原创最新推荐文章于 2022-01-12 13:40:56 发布 · 1.2k 阅读

0 ·

CC 4.0 BY-SA版权

本文为博主原创文章，未经博主允许不得转载。

解决问题专栏收录该内容

20 篇文章

订阅专栏

作者打算从hive - 0.14.0中剥离miniHS2，过程中抛出NullPointerException。曾在写Mapreduce时遇‘Could not locate executable null\\bin\\winutils.exe in the Hadoop binaries.’错误，起初想忽略，后发现问题仍由winutils.exe引起，还查看了方法执行栈。

今天打算从hive-0.14.0中剥离出一个miniHS2,但是在剥离的过程中抛出了一个NullPointerException,记一下排查问题的过程
下面是报错,几年前在写Mapreduce的时候,喷到过**Could not locate executable null\bin\winutils.exe in the Hadoop binaries.**这个错误,只需要配置一个winutils就可以了,但是想了一下是不是可以忽略这个报错呢,就没去管,最后发现还是因为winutils.exe引起的,不过也整好了解了一下为什么需要winutils。

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2065)
	at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:332)
	at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:95)
	at org.apache.hive.jdbc.LevinTest.main(LevinTest.java:33)
2019-07-12 10:38:48,505 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NullPointerException
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
	at org.apache.hadoop.util.Shell.run(Shell.java:455)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)
	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:472)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:597)
	at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:192)
	at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:220)
	at org.apache.hive.jdbc.miniHS2.MiniHS2.<init>(MiniHS2.java:216)
	at org.apache.hive.jdbc.LevinTest.main(LevinTest.java:35)
Disconnected from the target VM, address: '127.0.0.1:63904', transport: 'socket'

看了一下方法执行栈,是从下面的类开始的

 MiniHS2.java->  FileSystem.mkdirs(fs, wareHouseDir, FULL_PERM); //创建warehouse路径,并且授权

 FileSystem.java ->  // 问题出在了fs.setPermission(dir, permission); 中
 public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
  throws IOException {
    // create the directory using the default permission
    boolean result = fs.mkdirs(dir);
    // set its permission to be the supplied one
    fs.setPermission(dir, permission);
    return result;
  }
  
  RawLocalFileSystem.java -> // Shell.getSetPermissionCommand()方法会返回一个数组,问题就在这个
   public void setPermission(Path p, FsPermission permission)
    throws IOException {
    if (NativeIO.isAvailable()) {
      NativeIO.POSIX.chmod(pathToFile(p).getCanonicalPath(),
                     permission.toShort());
    } else {
      String perm = String.format("%04o", permission.toShort());
      Shell.execCommand(Shell.getSetPermissionCommand(perm, false,
        FileUtil.makeShellPath(pathToFile(p), true)));
    }
  }
  
  Shell.java ->  //继续跟到getSetPermissionCommand()中,马上就见到真像了
    public static String[] getSetPermissionCommand(String perm, boolean recursive,
                                                 String file) {
    String[] baseCmd = getSetPermissionCommand(perm, recursive);
    String[] cmdWithFile = Arrays.copyOf(baseCmd, baseCmd.length + 1);
    cmdWithFile[cmdWithFile.length - 1] = file;
    return cmdWithFile;
  }
Shell.java ->  //下面代码很简单,如果是WINDOWS系统,在返回这个命令数组的时候会在[0]拼接一个WINUTILS路径,
但是我的环境下执行这个数组的结构变成了这样 new String {null,"chmod","-R","777"},有个可能还不是很清楚,继续往下走
  public static String[] getSetPermissionCommand(String perm, boolean recursive) {
    if (recursive) {
      return (WINDOWS) ? new String[] { WINUTILS, "chmod", "-R", perm }
                         : new String[] { "chmod", "-R", perm };
    } else {
      return (WINDOWS) ? new String[] { WINUTILS, "chmod", perm }
                       : new String[] { "chmod", perm };
    }
  }

ProcessBuilder.java ->  看到下面的代码一下就明白了,在之前的debug的时候发现数组结构是这样的
	[1] chmod
	[2] -R
	[3] 777
因为[0]是null,所以intellij就自动给忽略了,我也没有注意到
public Process start() throws IOException {
        ...
        for (String arg : cmdarray)
            if (arg == null)
                throw new NullPointerException();
        // Throws IndexOutOfBoundsException if command is empty
        String prog = cmdarray[0];
        ...
 }