IDEA写hiveContext报错HiveSessionState

最新推荐文章于 2025-01-27 15:02:06 发布

原创最新推荐文章于 2025-01-27 15:02:06 发布 · 654 阅读

0 ·

CC 4.0 BY-SA版权

大数据专栏收录该内容

2 篇文章

订阅专栏

在使用IDEA开发过程中，遇到运行HiveContext时抛出IllegalArgumentException，错误涉及SparkSession的初始化失败。异常堆栈显示问题出在创建HiveSessionState时，具体为NullPointer异常。问题可能源于Hadoop配置、Hive元数据目录权限或者环境变量设置不正确。解决方法包括检查HDFS配置、确保Hive元数据目录存在且可写，以及确认相关环境变量已设置。请尝试排查这些方面并修复问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

错误：
Exception in thread “main” java.lang.IllegalArgumentException: Error while instantiating ‘org.apache.spark.sql.hive.HiveSessionState’:
at org.apache.spark.sql.SparkSession $. o r g$ apache $s p a r k$ sql $S p a r k S e s s i o n$ $r e f l e c t (S p a r k S e s s i o n . s c a l a : 981) a t o r g . a p a c h e . s p a r k . s q l . S p a r k S e s s i o n . s e s s i o n S t a t e$ lzycompute(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:574)
at org.apache.spark.sql.SQLContext.table(SQLContext.scala:708)
at com.imooc.spark.HiveContextApp $. m a i n (H i v e C o n t e x t A p p . s c a l a : 18) a t c o m . i m o o c . s p a r k . H i v e C o n t e x t A p p . m a i n (H i v e C o n t e x t A p p . s c a l a) C a u s e d b y : j a v a . l a n g . r e f l e c t . I n v o c a t i o n T a r g e t E x c e p t i o n a t s u n . r e f l e c t . N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e (N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e (D e l e g a t i n g C o n s t r u c t o r A c c e s s o r I m p l . j a v a : 45) a t j a v a . l a n g . r e f l e c t . C o n s t r u c t o r . n e w I n s t a n c e (C o n s t r u c t o r . j a v a : 423) a t o r g . a p a c h e . s p a r k . s q l . S p a r k S e s s i o n$ .org $a p a c h e$ spark $s q l$ SparkSession $KaTeX parse error: Can't use function '$' in math mode at position 212: \dotsnal.SharedState$̲.org$apache$spa\dots$ reflect(SharedState.scala:169)
at org.apache.spark.sql.internal.SharedState.(SharedState.scala:86)
at org.apache.spark.sql.SparkSession $KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲sharedState$1.a\dots$ anonfun$sharedState $1 . a p p l y (S p a r k S e s s i o n . s c a l a : 101) a t s c a l a . O p t i o n . g e t O r E l s e (O p t i o n . s c a l a : 120) a t o r g . a p a c h e . s p a r k . s q l . S p a r k S e s s i o n . s h a r e d S t a t e$ lzycompute(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
at org.apache.spark.sql.internal.SessionState.(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.(HiveSessionState.scala:32)
… 11 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.internal.SharedState $. o r g$ apache $s p a r k$ sql $i n t e r n a l$ SharedState$ $r e f l e c t (S h a r e d S t a t e . s c a l a : 166) . . . 19 m o r e C a u s e d b y : j a v a . l a n g . r e f l e c t . I n v o c a t i o n T a r g e t E x c e p t i o n a t s u n . r e f l e c t . N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e 0 (N a t i v e M e t h o d) a t s u n . r e f l e c t . N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e (N a t i v e C o n s t r u c t o r A c c e s s o r I m p l . j a v a : 62) a t s u n . r e f l e c t . D e l e g a t i n g C o n s t r u c t o r A c c e s s o r I m p l . n e w I n s t a n c e (D e l e g a t i n g C o n s t r u c t o r A c c e s s o r I m p l . j a v a : 45) a t j a v a . l a n g . r e f l e c t . C o n s t r u c t o r . n e w I n s t a n c e (C o n s t r u c t o r . j a v a : 423) a t o r g . a p a c h e . s p a r k . s q l . h i v e . c l i e n t . I s o l a t e d C l i e n t L o a d e r . c r e a t e C l i e n t (I s o l a t e d C l i e n t L o a d e r . s c a l a : 264) a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e U t i l s$ .newClientForMetadata(HiveUtils.scala:366)
at org.apache.spark.sql.hive.HiveUtils $. n e w C l i e n t F o r M e t a d a t a (H i v e U t i l s . s c a l a : 270) a t o r g . a p a c h e . s p a r k . s q l . h i v e . H i v e E x t e r n a l C a t a l o g . < i n i t > (H i v e E x t e r n a l C a t a l o g . s c a l a : 65) . . . 24 m o r e C a u s e d b y : j a v a . l a n g . R u n t i m e E x c e p t i o n : j a v a . l a n g . N u l l P o i n t e r E x c e p t i o n a t o r g . a p a c h e . h a d o o p . h i v e . q l . s e s s i o n . S e s s i o n S t a t e . s t a r t (S e s s i o n S t a t e . j a v a : 522) a t o r g . a p a c h e . s p a r k . s q l . h i v e . c l i e n t . H i v e C l i e n t I m p l . < i n i t > (H i v e C l i e n t I m p l . s c a l a : 192) . . . 32 m o r e C a u s e d b y : j a v a . l a n g . N u l l P o i n t e r E x c e p t i o n a t j a v a . l a n g . P r o c e s s B u i l d e r . s t a r t (P r o c e s s B u i l d e r . j a v a : 1012) a t o r g . a p a c h e . h a d o o p . u t i l . S h e l l . r u n C o m m a n d (S h e l l . j a v a : 404) a t o r g . a p a c h e . h a d o o p . u t i l . S h e l l . r u n (S h e l l . j a v a : 379) a t o r g . a p a c h e . h a d o o p . u t i l . S h e l l$ ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:678)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:661)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
at org.apache.hadoop.fs.RawLocalFileSystem $R a w L o c a l F i l e S t a t u s . l o a d P e r m i s s i o n I n f o (R a w L o c a l F i l e S y s t e m . j a v a : 567) a t o r g . a p a c h e . h a d o o p . f s . R a w L o c a l F i l e S y s t e m$ RawLocalFileStatus.getPermission(RawLocalFileSystem.java:542)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
… 33 more

求大佬解决！！！