Spark：报错- Cannot create directory /tmp/hive/hadoop/. Name node is in safe mode.

最新推荐文章于 2022-05-21 19:26:19 发布
原创 最新推荐文章于 2022-05-21 19:26:19 发布 · 2.3k 阅读 · 3 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#Cannot create directory /tmp/h #  Name node is in safe mode.

本文介绍了在使用 Apache Spark 和 Hive 进行大数据处理时遇到的一个具体错误：无法创建目录，因为 NameNode 处于安全模式。文章详细解释了错误产生的原因，并提供了明确的解决步骤，即使用 `hdfs dfsadmin -safemode leave` 命令来关闭安全模式。

java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:989)
  at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:116)
  at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:115)
  at org.apache.spark.sql.DataFrameReader.<init>(DataFrameReader.scala:549)
  at org.apache.spark.sql.SparkSession.read(SparkSession.scala:613)
  ... 52 elided
Caused by: java.lang.reflect.InvocationTargetException: java.lang.IllegalArgumentException: Error while instantiating
 'org.apache.spark.sql.hive.HiveExternalCatalog':  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:986)
  ... 56 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatal
og':  at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:
169)  at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
  at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
  at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
  at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
  at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
  at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
  ... 61 more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException: java.lang.Runtim
eException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/hadoop/9e7e1f41-6d39-41d1-952e-0a8503396376. Name node is in safe mode.The reported blocks 0 needs additional 251 blocks to reach the threshold 0.9990 of total blocks 251.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the 
thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtoc
olServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMe
thod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)


  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:
166)  ... 69 more
Caused by: java.lang.reflect.InvocationTargetException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteExcep
tion(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/hadoop/9e7e1f41-6d39-41d1-952e-0a8503396376. Name node is in safe mode.The reported blocks 0 needs additional 251 blocks to reach the threshold 0.9990 of total blocks 251.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the 
thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtoc
olServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMe
thod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)


  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
  at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
  at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
  ... 74 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.S
afeModeException): Cannot create directory /tmp/hive/hadoop/9e7e1f41-6d39-41d1-952e-0a8503396376. Name node is in safe mode.The reported blocks 0 needs additional 251 blocks to reach the threshold 0.9990 of total blocks 251.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the 
thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtoc
olServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMe
thod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)


  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
  at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
  ... 82 more
Caused by: org.apache.hadoop.ipc.RemoteException: Cannot create directory /tmp/hive/hadoop/9e7e1f41-6d39-41d1-952e-0a
8503396376. Name node is in safe mode.The reported blocks 0 needs additional 251 blocks to reach the threshold 0.9990 of total blocks 251.
The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the 
thresholds have been reached. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1364)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:4216)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:4191)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:813)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtoc
olServerSideTranslatorPB.java:600) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMe
thod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)


  at org.apache.hadoop.ipc.Client.call(Client.java:1469)
  at org.apache.hadoop.ipc.Client.call(Client.java:1400)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
  at com.sun.proxy.$Proxy22.mkdirs(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.j
ava:539)  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
  at com.sun.proxy.$Proxy23.mkdirs(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2742)
  at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2713)
  at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870)
  at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866)
  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866)
  at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859)
  at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)
  at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:574)
  at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
  ... 83 more


解决办法：

查看报错信息，会有提示解决办法的，如下

 Use       "hdfs dfsadmin -safemode leave"    to turn safe mode off.


完美解决！！！

大数据平台安装实验: ZooKeeper、Kafka、Hadoop、Hbase、Hive、Scala、Spark、Storm
zyw2002的博客
03-26 3498
文章目录实验1：Hadoop大数据平台安装实验1. 实验目的2. 实验环境3. 实验过程3.1 虚拟机的搭建3.1.1 安装虚拟机3.1.2 基本linux命令3.2 准备工作3.2.1 关闭Selinux3.2.2 安装软件3.3.3 检查网卡是否开机自启3.3.4 修改hosts3.3.5 检查网络是否正常3.3.6 新建hadoop用户3.3.7生成ssh密钥并分发3.3.8 安装NTP服务3.3 安装MySQL3.3.1 安装3.3.2 测试3.4 安装JDK3.4.1 安装3.4.2 测试3.5
大数据电商数仓--记录各种奇奇怪怪的issue
FunnyPrince_的博客
10-14 1447
⚠️报错： hive on spark正常 org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 65727339-603a-4fca-9df2-2f9d30e4b4a5 ⚠️报错： hive格式化报错Er
启动hadoop程序报错:Cannot create directory /tmp/hive. Name node is in safe mode
Yuki的博客
06-19 2110
运行hadoop程序时报错 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive. Name node is in safe mode 在分布式文件系统启动的时候，开始的时候NameNode...
Hadoop "Cannot create directory .Name node is in safe mode."解决方案
chuowen140381的博客
07-21 1743
转载自：http://www.waitig.com/hadoop-name-node-is-in-safe-mode.html 在使用Hadoop建立文件的时候，出现“Cannot create directory /user/hadoop/input. Name node is i...
Spark执行spark-shell 报错 Cannot create directory /tmp/hive/root/xxx. Name node is in safe mode
一起记录GIS学习
11-06 1019
1. 问题： Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 20/11/05 21:03:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin
虚拟机中启动hadoop报出Cannot create directory /tmp/hive/root/c6cf18ba-fc23-4385..........
BigData_C的博客
05-21 889
原因分析：因为hadoop的安全模式导致dbeaver不能连接hive报错 解决办法：关闭安全模式，在hadoop的安装目录执行命令：bin/hadoop dfsadmin -safemode leave
mkdir: Cannot create directory /usr/spark. Name node is in safe mode.
qq_42321843的博客
03-07 1434
使用如下命令在HDFS文件系统中创建本实验的工作目录： /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/spark 时遇到问题无法创建目录 mkdir: Cannot create directory /usr/spark. Name node is in safe mode. 这是因为该节点处于安全模式 可以采用强制退出的安全模式命令 找到ha...
[hadoop@node1 hive]$ bin/hive which: no hbase in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/export/server/jdk1.8/bin:/export/server/flume/bin:/export/server/hadoop/bin:/export/server/hive/bin:/export/server/spark/bin:/export/server/spark/sbin:/root/myenv:/export/server/jdk/bin:/export/server/hadoop/bin:/export/server/hadoop/sbin:/export/server/hive/bin:/export/server/spark/bin:/export/server/spark/sbin:/home/hadoop/.local/bin:/home/hadoop/bin) Hive Session ID = dcb273c2-25d3-4bf5-baeb-07905e307750 Logging initialized using configuration in jar:file:/export/server/apache-hive-3.1.2-bin/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hadoop/dcb273c2-25d3-4bf5-baeb-07905e307750. Name node is in safe mode. The reported blocks 773 needs additional 1 blocks to reach the threshold 0.9990 of total blocks 775. The minimum number of live datanodes is not required. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:node1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1661) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1648) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3530) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1173) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:750) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:621) at org.a
08-06
嗯，用户遇到了Hive启动失败的问题，报错信息明确指向HDFS安全模式（Name node is in safe mode）。从引用内容看，至少有四个来源都描述了相同现象：当HDFS处于安全模式时，Hive无法在/tmp/hive目录创建文件。用户的...
org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/root/fdc70186-83b6-4830-b6e1-7534b8f56ba3. Name node is in safe mode.
06-09
这个错误提示意味着你的Hadoop集群处于安全模式，无法创建指定目录。Hadoop进入安全模式通常是由于某些数据节点发生故障或者数据... hdfs dfsadmin -safemode leave 在退出安全模式后，你应该可以正常创建指定目录。
Restart App Timeline Server无法重启
L
02-23 1662
集群版本 HDP-2.6.3.0 YARN的App Timeline Server无法启动，导致整个YARN无法启动，如图： 错误如下： stderr: /var/lib/ambari-agent/data/errors-688.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cac...
错误提示：“put: Cannot create directory . Name node is in safe mode.”
hello world!
03-02 3374
错误提示 将本地文件复制到hdfs上去或者在hafs上新建文件时会出现“Cannot … directory … Name node is in safe mode”的错误。 例如 hadoop@master:~$ hadoop fs -put ~/data_test_spark yj_test put: Cannot create directory /user/hadoop/yj_tes...
启动hive报SafeModeException的错误
龙神add-走向数据架构师的路
06-07 1316
hive启动报错:hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp/hive/hadoop/676cc92f-18b0-4ca9-a1c6-80b061d7e974. Name node is in safe mode.
CDH spark submit error：Can't create directory Permission denied
星辰大数据的博客
04-27 1929
问题描述 cdh系统重新配置之后，发现spark任务提交失败，报错信息是本地目录创建的权限问题 Can't create directory Permission denied 问题解决 在yarn配置中搜索 cgroup，把“始终使用 Linux Container Executor”选项取消勾选，再重新部署配置 ...
SparkSQL创建数据库报错
gx304419380的博客
02-07 4789
错误信息 org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to create database path file:/D:/Java_Tools/spark/bin/spark-warehouse/tes...
spark：Failed to create local dir
自渡
10-06 2951
近日莫名遭遇异常一枚，如下： org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 271.0 failed 1 times, most recent failure: Lost task 0.0 in stage 271.0 (TID 544, localhost): java.io.IOException: Failed to create local dir in /tmp/blo...
spark报错Error while instantiating 'org.apache.spark.sql.internal.SessionState'
行路者
04-10 3852
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.internal.SessionState'…………………… Caused by: java.lang.reflect.InvocationTargetException ...
SparkSql整合Hive,使用idea运行,启动时报:"org.apache.spark.sql.hive.HiveSessionStateBuilder"
04-14 5184
异常错误 java.lang.IllegalArgumentException: Error while instantiating ‘org.apache.spark.sql.hive.HiveSessionStateBuilder’: 原因 Could not locate executable null\bin\winutils.exe in the Hadoop binaries 出现...
Spark-HBase集成错误之 java.lang.NoClassDefFoundError: org/htrace/Trace
weixin_33811539的博客
04-28 334
在进行Spark与HBase 集成的过程中遇到以下问题： java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': at org.apache.spark.sql.SparkSession$.org$apache...
IDEA通过SparkSql访问hive报错：Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder
zajbetterme的博客
03-25 7192
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': at org.apache.spark.sql.SparkSession$.org$apache$spark...
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot create directory /tmp/hive/atguigu/70264a9f-93d9-4e6f-bcac-c10db940a55c. Name node is in safe mode.
最新发布
11-20
当 NameNode 处于安全模式时，无法创建目录，因为安全模式是 HDFS 的一种保护机制，在该模式下，NameNode 只允许读取操作，不允许进行写操作（如创建目录）。以下是一些解决该问题的方法： ### 等待安全模式自动退出 NameNode 进入安全模式通常是在启动时检查文件系统的完整性。当检查完成且满足一定条件时，安全模式会自动退出。可以等待一段时间，让 NameNode 完成检查并自动退出安全模式。可以使用以下命令查看安全模式状态： ```bash hdfs dfsadmin -safemode get ``` ### 手动强制退出安全模式 如果确定文件系统已经稳定，可以手动强制 NameNode 退出安全模式。使用以下命令： ```bash hdfs dfsadmin -safemode leave ``` 执行此命令后，NameNode 会立即退出安全模式，之后就可以进行创建目录等写操作。 ### 检查数据块副本 安全模式可能因为数据块副本不足而持续存在。可以使用以下命令检查数据块副本情况： ```bash hdfs fsck / ``` 该命令会检查整个文件系统的数据块状态。如果发现有数据块副本不足的情况，HDFS 会尝试复制这些数据块以满足副本数量要求。 ### 检查集群健康状态 确保 Hadoop 集群中的所有 DataNode 都正常运行。可以使用以下命令查看 DataNode 状态： ```bash hdfs dfsadmin -report ``` 如果有 DataNode 处于异常状态，需要及时排查并修复问题。 ### 示例代码（Python 调用 HDFS 命令） ```python import subprocess # 检查安全模式状态 check_safemode_command = "hdfs dfsadmin -safemode get" result = subprocess.run(check_safemode_command, shell=True, capture_output=True, text=True) print(result.stdout) # 手动退出安全模式 leave_safemode_command = "hdfs dfsadmin -safemode leave" subprocess.run(leave_safemode_command, shell=True) ```
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值