window下Sparksql 无法获取文件权限问题

最新推荐文章于 2024-04-05 01:00:00 发布

原创

最新推荐文章于 2024-04-05 01:00:00 发布 · 1.1k 阅读

3 ·

CC 4.0 BY-SA版权

这篇博客介绍了在Windows环境下，由于Spark初始化Hive上下文时缺少winutils.exe导致的错误及其解决方案。首先，需要从指定链接下载winutils.exe和Hadoop，并将winutils.exe放入Hadoop的bin目录。接着，确保Hadoop的存储位置与报错信息中的路径一致，并设置环境变量HADOOP_HOME指向Hadoop根目录，将%HADOOP_HOME%in添加到Path中。最后，通过CMD执行相关命令修改tmphive目录的权限。完成这些步骤后，重启IDE并运行程序，问题通常可以得到解决。

Caused by: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F D:\tmp\hive

Caused by: java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F D:\tmp\hive
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:762)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:859)
	at org.apache.hadoop.util.Shell.execCommand(Shell.java:842)
	at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:587)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:562)
	at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
	at org.apache.spark.sql.hive.cl