Spark1.4 和 Hive 1.1.1 启动错误

最新推荐文章于 2024-06-10 23:34:36 发布

原创最新推荐文章于 2024-06-10 23:34:36 发布 · 2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hive #Spark

Spark 同时被 2 个专栏收录

21 篇文章

订阅专栏

Hive

19 篇文章

订阅专栏

本文详细记录了在使用Spark时遇到的问题：无法成功启动Spark SQL和Hive会话，具体表现为输入错误参数导致的NumberFormatException异常。通过排查发现参数格式不正确，最终修正并成功启动了Spark SQL环境，提供了调试过程和解决方案。

启动Spark-sql

[jifeng@feng03 spark-1.4.0-bin-hadoop2.6]$ ./bin/spark-sql  --master spark://feng03:7077
Exception in thread "main" java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "1s"
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:109)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NumberFormatException: For input string: "1s"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 11 more
15/09/05 14:47:31 INFO util.Utils: Shutdown hook called
15/09/05 14:47:31 INFO util.Utils: Deleting directory /tmp/spark-3fabb880-f749-41d9-9a4f-09e704e65cf3
[jifeng@feng03 spark-1.4.0-bin-hadoop2.6]$ ./bin/spark-sql --master spark://feng03:7077 --driver-class-path /home/jifeng/hive/mysql-connector-java-5.1.34.jar
Exception in thread "main" java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "1s"
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:109)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NumberFormatException: For input string: "1s"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 11 more
15/09/05 14:48:18 INFO util.Utils: Shutdown hook called
15/09/05 14:48:18 INFO util.Utils: Deleting directory /tmp/spark-72aa6ae8-5bba-437f-8636-fd0a2fe72d25

启动Spark-shell

[jifeng@feng03 hive]$ ./bin/spark-shell --master spark://feng03:7077 --driver-class-path /home/jifeng/hive/mysql-connector-java-5.1.34.jar
-bash: ./bin/spark-shell: No such file or directory
[jifeng@feng03 hive]$ cd ..
[jifeng@feng03 ~]$ cd spark-1.4.0-bin-hadoop2.6/
[jifeng@feng03 spark-1.4.0-bin-hadoop2.6]$ ./bin/spark-shell --master spark://feng03:7077 --driver-class-path /home/jifeng/hive/mysql-connector-java-5.1.34.jar
15/09/05 14:50:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/05 14:50:29 INFO spark.SecurityManager: Changing view acls to: jifeng
15/09/05 14:50:29 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/09/05 14:50:29 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/09/05 14:50:29 INFO spark.HttpServer: Starting HTTP Server
15/09/05 14:50:30 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/09/05 14:50:30 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:44520
15/09/05 14:50:30 INFO util.Utils: Successfully started service 'HTTP class server' on port 44520.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/09/05 14:50:41 INFO spark.SparkContext: Running Spark version 1.4.0
15/09/05 14:50:41 INFO spark.SecurityManager: Changing view acls to: jifeng
15/09/05 14:50:41 INFO spark.SecurityManager: Changing modify acls to: jifeng
15/09/05 14:50:41 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/09/05 14:50:41 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/09/05 14:50:42 INFO Remoting: Starting remoting
15/09/05 14:50:42 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.110:45394]
15/09/05 14:50:42 INFO util.Utils: Successfully started service 'sparkDriver' on port 45394.
15/09/05 14:50:42 INFO spark.SparkEnv: Registering MapOutputTracker
15/09/05 14:50:43 INFO spark.SparkEnv: Registering BlockManagerMaster
15/09/05 14:50:43 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-21bdebec-d78f-4491-bd03-52defdf21f68/blockmgr-7206fb4f-673f-40dd-b8dd-3b178ee39938
15/09/05 14:50:43 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB
15/09/05 14:50:43 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-21bdebec-d78f-4491-bd03-52defdf21f68/httpd-8c3af065-b7fe-4fb4-a5bd-8dde0b236698
15/09/05 14:50:43 INFO spark.HttpServer: Starting HTTP Server
15/09/05 14:50:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/09/05 14:50:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:37858
15/09/05 14:50:43 INFO util.Utils: Successfully started service 'HTTP file server' on port 37858.
15/09/05 14:50:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator
15/09/05 14:50:44 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/09/05 14:50:44 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
15/09/05 14:50:44 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
15/09/05 14:50:44 INFO ui.SparkUI: Started SparkUI at http://192.168.0.110:4040
15/09/05 14:50:45 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@feng03:7077/user/Master...
15/09/05 14:50:45 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150905145045-0000
15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor added: app-20150905145045-0000/0 on worker-20150905101207-192.168.0.110-58490 (192.168.0.110:58490) with 1 cores
15/09/05 14:50:46 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20150905145045-0000/0 on hostPort 192.168.0.110:58490 with 1 cores, 512.0 MB RAM
15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor updated: app-20150905145045-0000/0 is now RUNNING
15/09/05 14:50:46 INFO client.AppClient$ClientActor: Executor updated: app-20150905145045-0000/0 is now LOADING
15/09/05 14:50:46 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57911.
15/09/05 14:50:46 INFO netty.NettyBlockTransferService: Server created on 57911
15/09/05 14:50:46 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/09/05 14:50:46 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.110:57911 with 267.3 MB RAM, BlockManagerId(driver, 192.168.0.110, 57911)
15/09/05 14:50:46 INFO storage.BlockManagerMaster: Registered BlockManager
15/09/05 14:50:47 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
15/09/05 14:50:47 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
15/09/05 14:50:50 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "1s"
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:105)
        at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
        at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
        at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
        at $iwC$$iwC.<init>(<console>:9)
        at $iwC.<init>(<console>:18)
        at <init>(<console>:20)
        at .<init>(<console>:24)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:130)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
        at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
        at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
        at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
        at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NumberFormatException: For input string: "1s"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:492)
        at java.lang.Integer.parseInt(Integer.java:527)
        at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
        at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 56 more

15/09/05 14:50:58 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.0.110:44198/user/Executor#934818115]) with ID 0
15/09/05 14:50:59 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.110:35246 with 267.3 MB RAM, BlockManagerId(0, 192.168.0.110, 35246)
<console>:10: error: not found: value sqlContext
       import sqlContext.implicits._
              ^
<console>:10: error: not found: value sqlContext
       import sqlContext.sql
              ^

scala> exit