Caused by: java.io.IOException: Permission denied at java.io.UnixFileSystem.createFileExclus

最新推荐文章于 2025-04-13 19:23:48 发布

易悠

最新推荐文章于 2025-04-13 19:23:48 发布

阅读量1.1w

点赞数 1

CC 4.0 BY-SA版权

分类专栏：错误记录大数据 Spark Hadoop 文章标签： spark spark-shel

本文链接：https://blog.youkuaiyun.com/yizheyouye/article/details/51027281

Spark 同时被 3 个专栏收录

48 篇文章

订阅专栏

大数据

21 篇文章

订阅专栏

Hadoop

18 篇文章

订阅专栏

本文记录了解决Spark在启动过程中遇到的权限问题的过程。通过调整Hadoop配置和文件系统权限，最终成功启动Spark并创建了SparkContext。适用于Spark与Hadoop环境配置及故障排查。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

[hjr@master bin]$ ./spark-shell
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true
16/03/31 13:56:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/31 13:56:13 INFO spark.SecurityManager: Changing view acls to: hjr
16/03/31 13:56:13 INFO spark.SecurityManager: Changing modify acls to: hjr
16/03/31 13:56:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/03/31 13:56:13 INFO spark.HttpServer: Starting HTTP Server
16/03/31 13:56:13 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/03/31 13:56:13 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:39368
16/03/31 13:56:13 INFO util.Utils: Successfully started service 'HTTP class server' on port 39368.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.1
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
16/03/31 13:56:22 INFO spark.SparkContext: Running Spark version 1.4.1
16/03/31 13:56:22 WARN spark.SparkConf: 
SPARK_CLASSPATH was detected (set to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --driver-class-path to augment the driver classpath
 - spark.executor.extraClassPath to augment the executor classpath

16/03/31 13:56:22 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/03/31 13:56:22 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/03/31 13:56:22 INFO spark.SecurityManager: Changing view acls to: hjr
16/03/31 13:56:22 INFO spark.SecurityManager: Changing modify acls to: hjr
16/03/31 13:56:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/03/31 13:56:23 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/03/31 13:56:23 INFO Remoting: Starting remoting
16/03/31 13:56:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.13.43:40169]
16/03/31 13:56:23 INFO util.Utils: Successfully started service 'sparkDriver' on port 40169.
16/03/31 13:56:23 INFO spark.SparkEnv: Registering MapOutputTracker
16/03/31 13:56:23 INFO spark.SparkEnv: Registering BlockManagerMaster
16/03/31 13:56:23 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-9e5228e8-d1d9-49ac-b187-0da05e0359e4/blockmgr-a77c56e4-b68d-42fa-95f9-c0e783daf7d0
16/03/31 13:56:23 INFO storage.MemoryStore: MemoryStore started with capacity 1069.1 MB
16/03/31 13:56:24 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-9e5228e8-d1d9-49ac-b187-0da05e0359e4/httpd-f9ee9698-2a94-4b33-b660-64cfa3e2e4f2
16/03/31 13:56:24 INFO spark.HttpServer: Starting HTTP Server
16/03/31 13:56:24 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/03/31 13:56:24 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52432
16/03/31 13:56:24 INFO util.Utils: Successfully started service 'HTTP file server' on port 52432.
16/03/31 13:56:24 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/03/31 13:56:24 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/03/31 13:56:24 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/03/31 13:56:24 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/03/31 13:56:24 INFO ui.SparkUI: Started SparkUI at http://192.168.13.43:4040
16/03/31 13:56:25 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@master.casking.com:7077/user/Master...
16/03/31 13:56:25 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160331135625-0015
16/03/31 13:56:25 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54303.
16/03/31 13:56:25 INFO netty.NettyBlockTransferService: Server created on 54303
16/03/31 13:56:25 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/03/31 13:56:25 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.43:54303 with 1069.1 MB RAM, BlockManagerId(driver, 192.168.13.43, 54303)
16/03/31 13:56:25 INFO storage.BlockManagerMaster: Registered BlockManager
16/03/31 13:56:26 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/03/31 13:56:26 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/03/31 13:56:28 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
java.lang.RuntimeException: java.io.IOException: Permission denied
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:330)
        at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:116)
        at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
        at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
        at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:168)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
        at $iwC$$iwC.<init>(<console>:9)
        at $iwC.<init>(<console>:18)
        at <init>(<console>:20)
        at .<init>(<console>:24)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:130)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
        at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
        at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
        at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
        at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2001)
        at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:432)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:328)
        ... 56 more

<console>:10: error: not found: value sqlContext
       import sqlContext.implicits._
              ^
<console>:10: error: not found: value sqlContext
       import sqlContext.sql
              ^

scala>

查看Hadoop的core-site.xml文件的hadoop.tmp.dir的配置路径，并
修改其访问权限：

 <property>
       <name>hadoop.tmp.dir</name>
       <value>file:/home/spark/hadoop-2.4.0/temp</value>
      <description>Abase for other temporary directories.    
      </description>
 </property>

我的spark和hadoop安装配置都是使用root用户，所以，将新建的用户hjr新加到root组里面去，然后再修改hadoop.tmp.dir路径的权限：

hadoop/spark集群安装配置在root组：

[root@master spark-1.4.1-bin-hadoop2.4]# id
uid=0(root) gid=0(root) groups=0(root)
[root@master spark-1.4.1-bin-hadoop2.4]#

查看hjr用户所属组：

[root@master spark-1.4.1-bin-hadoop2.4]# su hjr
[hjr@master spark-1.4.1-bin-hadoop2.4]$ id
uid=1012(hjr) gid=1012(hjr) groups=1012(hjr)
[hjr@master spark-1.4.1-bin-hadoop2.4]$

将用户hjr添加到root组里面去：

[hjr@master spark-1.4.1-bin-hadoop2.4]$ su root
Password: 
[root@master spark-1.4.1-bin-hadoop2.4]# usermod -g root hjr
[root@master spark-1.4.1-bin-hadoop2.4]# su hjr
[hjr@master spark-1.4.1-bin-hadoop2.4]$ id
uid=1012(hjr) gid=0(root) groups=0(root)
[hjr@master spark-1.4.1-bin-hadoop2.4]$

退出spark-shell来,发现问题了：

根目录下的/tmp文件夹的访问权限没有提上来：

scala> exit
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
16/04/06 14:12:40 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/04/06 14:12:40 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.13.43:4040
16/04/06 14:12:40 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/04/06 14:12:40 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors
16/04/06 14:12:40 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down
16/04/06 14:12:40 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/06 14:12:40 INFO util.Utils: path = /tmp/spark-4321496c-dcdc-4d6f-abe2-576d81b4a2fb/blockmgr-44be0eb3-a333-4f63-827d-05c926a3d3d1, already present as root for deletion.
16/04/06 14:12:40 INFO storage.MemoryStore: MemoryStore cleared
16/04/06 14:12:40 INFO storage.BlockManager: BlockManager stopped
16/04/06 14:12:40 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/04/06 14:12:40 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/06 14:12:40 INFO spark.SparkContext: Successfully stopped SparkContext
16/04/06 14:12:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/04/06 14:12:40 INFO util.Utils: Shutdown hook called
16/04/06 14:12:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-4321496c-dcdc-4d6f-abe2-576d81b4a2fb
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-b177a1a8-0c25-4475-8f06-dddb3a49fe08
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-f731a0a4-c5cf-4db5-baab-ec3a4d28cf26

切换到root用户下，将/tmp文件夹权限提上来就行：

chmod -R 777 /tmp

再次执行，妥妥的。。。。。。

[hjr@master bin]$ ./spark-shell
16/04/06 15:07:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/06 15:07:39 INFO spark.SecurityManager: Changing view acls to: hjr
16/04/06 15:07:39 INFO spark.SecurityManager: Changing modify acls to: hjr
16/04/06 15:07:39 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/04/06 15:07:40 INFO spark.HttpServer: Starting HTTP Server
16/04/06 15:07:40 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:40 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:51949
16/04/06 15:07:40 INFO util.Utils: Successfully started service 'HTTP class server' on port 51949.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.1
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
16/04/06 15:07:49 INFO spark.SparkContext: Running Spark version 1.4.1
16/04/06 15:07:49 WARN spark.SparkConf: 
SPARK_CLASSPATH was detected (set to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --driver-class-path to augment the driver classpath
 - spark.executor.extraClassPath to augment the executor classpath

16/04/06 15:07:49 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/04/06 15:07:49 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/04/06 15:07:49 INFO spark.SecurityManager: Changing view acls to: hjr
16/04/06 15:07:49 INFO spark.SecurityManager: Changing modify acls to: hjr
16/04/06 15:07:49 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/04/06 15:07:50 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/04/06 15:07:50 INFO Remoting: Starting remoting
16/04/06 15:07:50 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.13.43:59810]
16/04/06 15:07:50 INFO util.Utils: Successfully started service 'sparkDriver' on port 59810.
16/04/06 15:07:50 INFO spark.SparkEnv: Registering MapOutputTracker
16/04/06 15:07:50 INFO spark.SparkEnv: Registering BlockManagerMaster
16/04/06 15:07:50 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-eb699480-b70b-4273-b8e9-433764a62c0e/blockmgr-22f931fa-c02a-409e-8acc-af6b4ce3f309
16/04/06 15:07:50 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB
16/04/06 15:07:51 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-eb699480-b70b-4273-b8e9-433764a62c0e/httpd-d5ad4baa-989e-4731-9944-652645c86fef
16/04/06 15:07:51 INFO spark.HttpServer: Starting HTTP Server
16/04/06 15:07:51 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:51 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:47726
16/04/06 15:07:51 INFO util.Utils: Successfully started service 'HTTP file server' on port 47726.
16/04/06 15:07:51 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/04/06 15:07:51 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:51 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/04/06 15:07:51 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/04/06 15:07:51 INFO ui.SparkUI: Started SparkUI at http://192.168.13.43:4040
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@master.casking.com:7077/user/Master...
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160406150752-0004
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor added: app-20160406150752-0004/0 on worker-20160406135801-192.168.13.45-47341 (192.168.13.45:47341) with 1 cores
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160406150752-0004/0 on hostPort 192.168.13.45:47341 with 1 cores, 512.0 MB RAM
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor added: app-20160406150752-0004/1 on worker-20160406135801-192.168.13.44-60023 (192.168.13.44:60023) with 1 cores
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160406150752-0004/1 on hostPort 192.168.13.44:60023 with 1 cores, 512.0 MB RAM
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/1 is now LOADING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/0 is now LOADING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/0 is now RUNNING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/1 is now RUNNING
16/04/06 15:07:53 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51708.
16/04/06 15:07:53 INFO netty.NettyBlockTransferService: Server created on 51708
16/04/06 15:07:53 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/04/06 15:07:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.43:51708 with 267.3 MB RAM, BlockManagerId(driver, 192.168.13.43, 51708)
16/04/06 15:07:53 INFO storage.BlockManagerMaster: Registered BlockManager
16/04/06 15:07:53 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/04/06 15:07:53 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/04/06 15:07:55 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
16/04/06 15:07:56 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.13.43:9083
16/04/06 15:07:56 INFO hive.metastore: Connected to metastore.
16/04/06 15:07:57 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.13.45:51876/user/Executor#820989273]) with ID 0
16/04/06 15:07:57 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.13.44:38556/user/Executor#-638549016]) with ID 1
16/04/06 15:07:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.44:35787 with 267.3 MB RAM, BlockManagerId(1, 192.168.13.44, 35787)
16/04/06 15:07:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.45:33428 with 267.3 MB RAM, BlockManagerId(0, 192.168.13.45, 33428)
16/04/06 15:07:58 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/04/06 15:07:58 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

scala>