windows部署spark:The root scratch dir: /tmp/hive on HDFS should be writable.

最新推荐文章于 2024-03-04 10:03:56 发布

YanfangDu

最新推荐文章于 2024-03-04 10:03:56 发布

阅读量902

点赞数

分类专栏： bigdata

本文链接：https://blog.youkuaiyun.com/dfyBlog/article/details/99974247

版权

1.安装jdk

配置环境变量

JAVA_HOME:D:\java\jdk1.8.0_31
PATH:%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin
CLASSPATH:.;%JAVA_HOME%\lib;%JAVA_HOME%\lib\tools.jar

注:jdk尽量安装到自定义目录,默认的Program File有空格,可能会导致后续运行某些命令报错

2.安装hadoop

下载:http://archive.apache.org/dist/hadoop/core/

环境变量配置

HADOOP_HOME=D:\java\hadoop-2.8.4
PATH:%HADOOP_HOME%\bin

详细参考:https://blog.youkuaiyun.com/luanpeng825485697/article/details/79420532

3.安装Scala

下载地址:http://www.scala-lang.org/download/all.html
这里使用的是2.11.8版本的

配置环境变量

SCALA_HOME:D:\java\scala-2.11.8
PATH:%SCALA_HOME%\bin

验证是否安装成功:
在cmd窗口中，输入 scala，回车,如下图

4.安装spark

地址:http://spark.apache.org/downloads.html

配置环境变量

SPARK_HOME:D:\java\spark-2.1.0-bin-hadoop2.7
Path:%SPARK_HOME%\bin

cmd 运行spark-shell
- 自

最低0.47元/天解锁文章

确定要放弃本次机会？

福利倒计时

: :

立减 ¥

普通VIP年卡可用

立即使用

YanfangDu

关注关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
分享

复制链接

分享到 QQ

分享到新浪微博

扫一扫
举报

举报

专栏目录

Hive填坑记1：win10下报The root scratch dir: /tmp/hive on HDFS should be writable

sunlen的专栏(编程技术探讨)

01-16

970

在windows环境下，跑Spark程序，报“tmp/hive”权限不对，用docs命令加上权限，又提示“ChangeFileModeByMask error”，网上没有一个正确的解决方案，几经折腾，最终搞定，经验分享给大家。我在window10跑Spark离线任务，抛如下异常： Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lan...

HIVE启动错误：The root scratch dir: hdfs://master:8020/user/hive/tmp on HDFS should be writable的详细解决方法。

m0_55969961的博客

11-20

1410

详细步骤解决：Exception in thread "main" java.lang.RuntimeException: The root scratch dir: hdfs://mast/user/hive/tmp on HDFS should be writable. Current permissions are: rwxr-xr-x

参与评论您还未登录，请先登录后发表或查看评论

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x

成长加速度

01-10

540

启动hive报的异常使用hadoop fs -chmod -R 777 /tmp修改目录权限

【报错】Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writabl

最新发布

weixin_61070671的博客

03-04

1165

【代码】【报错】Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writabl。

windows安装spark出现The root scratch dir: /tmp/hive on HDFS should be writable

bhcgdh的博客

06-19

3588

我安装的版本如下： Scala 2.11.8 java version “1.8.0_171” spark-2.2.0-bin-hadoop2.7 hadoop-2.7.6 另外安装了winutils到hadoop bin中* 启动spark-shell时报错，其中一个错误是说没有权限，错误如下： The root scratch dir: /tmp/hive on HDFS sh...

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are : rw-rw-rw-

weixin_33704234的博客

05-04

797

2019独角兽企业重金招聘Python工程师标准>>> ...

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-

zgjdzwhy的博客

05-01

8110

It is easy to follow the instructions on http://spark.apache.org/docs/latest/ and download Spark 1.6.0 (Jan 04 2016) with the “Pre-build for Hadoop 2.6 and later” package type from http://spark.apac

spark-shell连接异常问题

哈尔科夫坦克厂的播客

11-05

742

[root@node00 sbin]# /apps/spark-2.2.0-bin-hadoop2.7/bin/spark-shell --master spark://node00:7077 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log le...

在idea里面启动spark程序链接hive出现：The root scratch dir: /tmp/hive on HDFS should be writable. Current permiss

m0_55243062的博客

02-16

1148

The root scratch dir: /tmp/hive on HDFS should be writable.（已解决）

Han_Lin-的博客

12-15

577

The root scratch dir: /tmp/hive on HDFS should be writable.

解决windows上The root scratch dir: /tmp/hive on HDFS should be writable.Current permissions are: ------

heccxx的博客

11-02

3640

解决Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: --------- 首先...

在idea运行scala程序报错：The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are

qq_37023928的博客

05-10

735

在idea运行scala程序报错：Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: --------- 解决方式： %HADOOP_HOME%\bin\winutils.exe chmod 7

The root scratch dir: tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-

weixin_43995212的博客

06-17

750

The root scratch dir: tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-

解决 The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x

m0_52184330的博客

11-15

4690

问题描述： spark 将数据写入到hive表时报错： The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x 如下图：原因分析：从The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x 可以看出，系统要求再hdfs上用户要具备写的权

sparksql 读取、连接 hive 报The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions

qq_42044006的博客

03-02

2920

sparksql 读取、连接 hive 报The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions 首先检查权限问题，这里有两种解决方式： 1.用有写权限的用户去操作：在代码中添加环境变量：System.setProperty("HADOOP_USER_NAME","hadoop") 2.其次检查 hdfs /tmp/hive 其他用户是否对其有写的权限，如果没有就用该目录拥有者账户执行hadoop

Spark (null) entry in command string: null chmod 0700

tom_fans的博客

12-27

2843

使用ecplise来提交SPARK SQL的时候报错： Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.io.IOException: (null) entry in command string: null chmod 0700 C:\Use...

Hive启动出现权限错误 /tmp/hive on HDFS should be writable.

yamaxifeng_132的专栏

02-09

2022

Hive启动出现权限错误 /tmp/hive on HDFS should be writable.

hive启动出现权限错误 /tmp/hive on HDFS should be writable.

热门推荐

bear

04-23

1万+

启动hive时出现Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.Current permissions are: rwxr-xr-x

azkaban任务报错java.lang.RuntimeException: The root scratch dir: /tmp/hive

TURING.DT

04-13

3359

azkaban运行任务的时候失败报错如下： 23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: The root scratch d

D:\tmp\hive

07-28

回答: D:\tmp\hive是一个文件路径，根据引用\[1\]中的解决方法，可以通过修改winutils.exe的权限来操作该文件路径。首先，将winutils.exe和hadoop.dll文件拷贝到Windows下解压后的hadoop的bin文件夹下，并配置环境变量。然后，在命令行中进入winutils.exe所在目录，使用命令"winutils.exe chmod 777 D:\tmp\hive"来修改该文件路径的权限。最后，重启IDEA即可。 #### 引用[.reference_title] - *1* *3* [idea执行spark sql报错：java.io.IOException: (null) entry in command string: null ls -F D:\tmp\hive](https://blog.csdn.net/huazi99/article/details/124923822)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^insertT0,239^v3^insert_chatgpt"}} ] [.reference_item] - *2* [在idea运行spark程序报错：The root scratch dir: /tmp/hive on HDFS should be writable](https://blog.csdn.net/penriver/article/details/123248416)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v91^insertT0,239^v3^insert_chatgpt"}} ] [.reference_item] [ .reference_list ]