idea 中运行spark 保存到hive中，由于没有配置hive信息，spark运行会默认运行内部的hive...

最新推荐文章于 2024-04-07 11:05:09 发布

转载最新推荐文章于 2024-04-07 11:05:09 发布 · 298 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://my.oschina.net/u/3962987/blog/3086514

本文详细记录了解决Spark在连接Hive时遇到的异常问题，主要原因是未正确配置Hive信息导致。通过调整配置，确保Hive目录权限正确，最终成功解决了运行时出现的IllegalArgumentException错误。

原因：idea 中运行spark 保存到hive中，由于没有配置hive信息，spark运行会默认运行内部的hive

参考地址：（我尝试了，有效解决了）

https://blog.youkuaiyun.com/zgjdzwhy/article/details/71056801

http://mangocool.com/1473838702533.html

运行结果：Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
   at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
   at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
   at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
   at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
   at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
   at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
   at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
   at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
   at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878)
   at MockDataGenerate$.main(MockDataGenerate.scala:167)
   at MockDataGenerate.main(MockDataGenerate.scala)
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
   ... 12 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
   at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169)
   at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
   at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
   at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
   at scala.Option.getOrElse(Option.scala:121)
   at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
   at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
   at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
   at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
   ... 17 more
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
   ... 25 more
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
   at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
   at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
   at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
   ... 30 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-
   at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
   at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
   ... 38 more
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-
   at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
   at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
   at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
   ... 39 more

转载于:https://my.oschina.net/u/3962987/blog/3086514