spark sql执行失败

最新推荐文章于 2025-04-07 16:08:49 发布
转载 最新推荐文章于 2025-04-07 16:08:49 发布 · 3.4k 阅读 · 0
文章标签：

#spark #sql #spark-sql

1：运行 ./bin/spark-sql

需要先把hive-site.xml 负责到spark的conf目录下


[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql
Spark assembly has been built with Hive, including Datanucleus jars on classpath
java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties


提示编译的时候要带2个参数

重新编译：./make-distribution.sh --tgz -Phadoop-2.4 -Pyarn -DskipTests -Dhadoop.version=2.4.1 -Phive -Phive-thriftserver


2：再次运行：

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql   
Spark assembly has been built with Hive, including Datanucleus jars on classpath
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:101)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
        ... 14 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
        at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:587)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:788)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:333)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
        at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
        at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
        at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
        at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
        ... 19 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:325)
        at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:282)
        at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:240)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:286)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
        at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
        at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1187)
        at org.datanucleus.NucleusContext.initialise(NucleusContext.java:356)
        at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:775)
        ... 48 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "dbcp-builtin" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:259)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:131)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:85)
        ... 66 more
Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.
        at org.datanucleus.store.rdbms.connectionpool.AbstractConnectionPoolFactory.loadDriver(AbstractConnectionPoolFactory.java:58)
        at org.datanucleus.store.rdbms.connectionpool.DBCPBuiltinConnectionPoolFactory.createConnectionPool(DBCPBuiltinConnectionPoolFactory.java:49)
        at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:238)
        ... 68 more
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties


发现找不到mysql的jdbc包

3：加载jdbc运行

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar
Spark assembly has been built with Hive, including Datanucleus jars on classpath
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/03/04 14:30:01 INFO SecurityManager: Changing view acls to: jifeng
15/03/04 14:30:01 INFO SecurityManager: Changing modify acls to: jifeng
15/03/04 14:30:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/04 14:30:02 INFO Slf4jLogger: Slf4jLogger started
15/03/04 14:30:02 INFO Remoting: Starting remoting
15/03/04 14:30:02 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@feng02:53897]
15/03/04 14:30:02 INFO Utils: Successfully started service 'sparkDriver' on port 53897.
15/03/04 14:30:02 INFO SparkEnv: Registering MapOutputTracker
15/03/04 14:30:02 INFO SparkEnv: Registering BlockManagerMaster
15/03/04 14:30:02 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150304143002-0534
15/03/04 14:30:02 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
15/03/04 14:30:02 INFO HttpFileServer: HTTP File server directory is /tmp/spark-5172cadb-24ba-4eaa-b78b-cefb2e0ffb5b
15/03/04 14:30:02 INFO HttpServer: Starting HTTP Server
15/03/04 14:30:02 INFO Utils: Successfully started service 'HTTP file server' on port 55683.
15/03/04 14:30:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/03/04 14:30:03 INFO SparkUI: Started SparkUI at http://feng02:4040
15/03/04 14:30:03 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@feng02:53897/user/HeartbeatReceiver
15/03/04 14:30:03 INFO NettyBlockTransferService: Server created on 39062
15/03/04 14:30:03 INFO BlockManagerMaster: Trying to register BlockManager
15/03/04 14:30:03 INFO BlockManagerMasterActor: Registering block manager localhost:39062 with 267.3 MB RAM, BlockManagerId(<driver>, localhost, 39062)
15/03/04 14:30:03 INFO BlockManagerMaster: Registered BlockManager
SET spark.sql.hive.version=0.13.1
15/03/04 14:30:04 INFO HiveMetaStore: 0: get_all_databases
15/03/04 14:30:04 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_all_databases
15/03/04 14:30:04 INFO HiveMetaStore: 0: get_functions: db=default pat=*
15/03/04 14:30:04 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_functions: db=default pat=*
15/03/04 14:30:04 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
spark-sql>


4：操作：

show tables;

spark-sql> show tables;
15/03/04 14:39:23 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/04 14:39:23 INFO ParseDriver: Parsing command: show tables
15/03/04 14:39:23 INFO ParseDriver: Parse Completed
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO Driver: Concurrency mode is disabled, not creating a lock manager
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO ParseDriver: Parsing command: show tables
15/03/04 14:39:24 INFO ParseDriver: Parse Completed
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=parse start=1425451164380 end=1425451164381 duration=1 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO Driver: Semantic Analysis Completed
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=semanticAnalyze start=1425451164381 end=1425451164458 duration=77 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO ListSinkOperator: Initializing Self 0 OP
15/03/04 14:39:24 INFO ListSinkOperator: Operator 0 OP initialized
15/03/04 14:39:24 INFO ListSinkOperator: Initialization Done 0 OP
15/03/04 14:39:24 INFO Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=compile start=1425451164361 end=1425451164622 duration=261 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO Driver: Starting command: show tables
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=TimeToSubmit start=1425451164358 end=1425451164625 duration=267 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO HiveMetaStore: 0: get_database: default
15/03/04 14:39:24 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_database: default
15/03/04 14:39:24 INFO HiveMetaStore: 0: get_tables: db=default pat=.*
15/03/04 14:39:24 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_tables: db=default pat=.*
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=runTasks start=1425451164625 end=1425451164660 duration=35 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=Driver.execute start=1425451164622 end=1425451164660 duration=38 from=org.apache.hadoop.hive.ql.Driver>
OK
15/03/04 14:39:24 INFO Driver: OK
15/03/04 14:39:24 INFO PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=releaseLocks start=1425451164660 end=1425451164660 duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO PerfLogger: </PERFLOG method=Driver.run start=1425451164358 end=1425451164660 duration=302 from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:24 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/03/04 14:39:24 INFO FileInputFormat: Total input paths to process : 1
15/03/04 14:39:24 INFO ListSinkOperator: 0 finished. closing...
15/03/04 14:39:24 INFO ListSinkOperator: 0 Close done
15/03/04 14:39:24 INFO SparkContext: Starting job: collect at SparkPlan.scala:84
15/03/04 14:39:24 INFO DAGScheduler: Got job 0 (collect at SparkPlan.scala:84) with 1 output partitions (allowLocal=false)
15/03/04 14:39:24 INFO DAGScheduler: Final stage: Stage 0(collect at SparkPlan.scala:84)
15/03/04 14:39:24 INFO DAGScheduler: Parents of final stage: List()
15/03/04 14:39:24 INFO DAGScheduler: Missing parents: List()
15/03/04 14:39:24 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[2] at map at SparkPlan.scala:84), which has no missing parents
15/03/04 14:39:25 INFO MemoryStore: ensureFreeSpace(2560) called with curMem=0, maxMem=280248975
15/03/04 14:39:25 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.5 KB, free 267.3 MB)
15/03/04 14:39:25 INFO MemoryStore: ensureFreeSpace(1561) called with curMem=2560, maxMem=280248975
15/03/04 14:39:25 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1561.0 B, free 267.3 MB)
15/03/04 14:39:25 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:39062 (size: 1561.0 B, free: 267.3 MB)
15/03/04 14:39:25 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
15/03/04 14:39:25 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:838
15/03/04 14:39:25 INFO DAGScheduler: Submitting 1 missing tasks from Stage 0 (MappedRDD[2] at map at SparkPlan.scala:84)
15/03/04 14:39:25 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/03/04 14:39:25 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1378 bytes)
15/03/04 14:39:25 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
15/03/04 14:39:25 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 744 bytes result sent to driver
15/03/04 14:39:25 INFO DAGScheduler: Stage 0 (collect at SparkPlan.scala:84) finished in 0.202 s
15/03/04 14:39:25 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 191 ms on localhost (1/1)
15/03/04 14:39:25 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/03/04 14:39:25 INFO DAGScheduler: Job 0 finished: collect at SparkPlan.scala:84, took 0.625512 s
15/03/04 14:39:25 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@a440292
15/03/04 14:39:25 INFO StatsReportListener: task runtime:(count: 1, mean: 191.000000, stdev: 0.000000, max: 191.000000, min: 191.000000)
course
hbase_table_1
pokes
student
Time taken: 2.535 seconds
15/03/04 14:39:25 INFO CliDriver: Time taken: 2.535 seconds
15/03/04 14:39:25 INFO PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/03/04 14:39:25 INFO PerfLogger: </PERFLOG method=releaseLocks start=1425451165520 end=1425451165520 duration=0 from=org.apache.hadoop.hive.ql.Driver>
spark-sql> 15/03/04 14:39:25 INFO StatsReportListener:  0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:39:25 INFO StatsReportListener:     191.0 ms        191.0 ms        191.0 ms        191.0 ms        191.0 ms191.0 ms        191.0 ms        191.0 ms        191.0 ms
15/03/04 14:39:25 INFO StatsReportListener: task result size:(count: 1, mean: 744.000000, stdev: 0.000000, max: 744.000000, min: 744.000000)
15/03/04 14:39:25 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:39:25 INFO StatsReportListener:     744.0 B 744.0 B 744.0 B 744.0 B 744.0 B 744.0 B 744.0 B 744.0 B 744.0 B
15/03/04 14:39:25 INFO StatsReportListener: executor (non-fetch) time pct: (count: 1, mean: 8.900524, stdev: 0.000000, max: 8.900524, min: 8.900524)
15/03/04 14:39:25 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:39:25 INFO StatsReportListener:      9 %     9 %     9 %     9 %     9 %     9 %     9 %     9 %     9 %
15/03/04 14:39:25 INFO StatsReportListener: other time pct: (count: 1, mean: 91.099476, stdev: 0.000000, max: 91.099476, min: 91.099476)
15/03/04 14:39:25 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:39:25 INFO StatsReportListener:     91 %    91 %    91 %    91 %    91 %    91 %    91 %    91 %    91 %


select * from student;

> select * from student;
15/03/04 14:42:05 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/04 14:42:05 INFO ParseDriver: Parsing command: select * from student
15/03/04 14:42:05 INFO ParseDriver: Parse Completed
15/03/04 14:42:05 INFO HiveMetaStore: 0: get_table : db=default tbl=student
15/03/04 14:42:05 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_table : db=default tbl=student
15/03/04 14:42:05 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/03/04 14:42:05 INFO MemoryStore: ensureFreeSpace(444571) called with curMem=4121, maxMem=280248975
15/03/04 14:42:05 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 434.2 KB, free 266.8 MB)
15/03/04 14:42:05 INFO MemoryStore: ensureFreeSpace(47070) called with curMem=448692, maxMem=280248975
15/03/04 14:42:05 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 46.0 KB, free 266.8 MB)
15/03/04 14:42:05 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:39062 (size: 46.0 KB, free: 267.2 MB)
15/03/04 14:42:05 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/03/04 14:42:05 INFO SparkContext: Created broadcast 1 from broadcast at TableReader.scala:68
15/03/04 14:42:06 INFO BlockManager: Removing broadcast 0
15/03/04 14:42:06 INFO BlockManager: Removing block broadcast_0
15/03/04 14:42:06 INFO MemoryStore: Block broadcast_0 of size 2560 dropped from memory (free 279755773)
15/03/04 14:42:06 INFO BlockManager: Removing block broadcast_0_piece0
15/03/04 14:42:06 INFO MemoryStore: Block broadcast_0_piece0 of size 1561 dropped from memory (free 279757334)
15/03/04 14:42:06 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:39062 in memory (size: 1561.0 B, free: 267.2 MB)
15/03/04 14:42:06 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
15/03/04 14:42:06 INFO ContextCleaner: Cleaned broadcast 0
15/03/04 14:42:06 INFO FileInputFormat: Total input paths to process : 1
15/03/04 14:42:06 INFO SparkContext: Starting job: collect at SparkPlan.scala:84
15/03/04 14:42:06 INFO DAGScheduler: Got job 1 (collect at SparkPlan.scala:84) with 2 output partitions (allowLocal=false)
15/03/04 14:42:06 INFO DAGScheduler: Final stage: Stage 1(collect at SparkPlan.scala:84)
15/03/04 14:42:06 INFO DAGScheduler: Parents of final stage: List()
15/03/04 14:42:06 INFO DAGScheduler: Missing parents: List()
15/03/04 14:42:06 INFO DAGScheduler: Submitting Stage 1 (MappedRDD[7] at map at SparkPlan.scala:84), which has no missing parents
15/03/04 14:42:06 INFO MemoryStore: ensureFreeSpace(8616) called with curMem=491641, maxMem=280248975
15/03/04 14:42:06 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 8.4 KB, free 266.8 MB)
15/03/04 14:42:06 INFO MemoryStore: ensureFreeSpace(5184) called with curMem=500257, maxMem=280248975
15/03/04 14:42:06 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.1 KB, free 266.8 MB)
15/03/04 14:42:06 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:39062 (size: 5.1 KB, free: 267.2 MB)
15/03/04 14:42:06 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/03/04 14:42:06 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
15/03/04 14:42:06 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MappedRDD[7] at map at SparkPlan.scala:84)
15/03/04 14:42:06 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
15/03/04 14:42:06 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, ANY, 1318 bytes)
15/03/04 14:42:06 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
15/03/04 14:42:06 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/student/stu.txt:0+28
15/03/04 14:42:06 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
15/03/04 14:42:06 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
15/03/04 14:42:06 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
15/03/04 14:42:06 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
15/03/04 14:42:06 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
15/03/04 14:42:06 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1843 bytes result sent to driver
15/03/04 14:42:06 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, localhost, ANY, 1318 bytes)
15/03/04 14:42:06 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
15/03/04 14:42:06 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/student/stu.txt:28+29
15/03/04 14:42:06 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 267 ms on localhost (1/2)
15/03/04 14:42:06 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1828 bytes result sent to driver
15/03/04 14:42:06 INFO DAGScheduler: Stage 1 (collect at SparkPlan.scala:84) finished in 0.313 s
15/03/04 14:42:06 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@61f0612d
15/03/04 14:42:06 INFO DAGScheduler: Job 1 finished: collect at SparkPlan.scala:84, took 0.338182 s
15/03/04 14:42:06 INFO StatsReportListener: task runtime:(count: 2, mean: 165.000000, stdev: 102.000000, max: 267.000000, min: 63.000000)
15/03/04 14:42:06 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:42:06 INFO StatsReportListener:     63.0 ms 63.0 ms 63.0 ms 63.0 ms 267.0 ms        267.0 ms        267.0 ms267.0 ms        267.0 ms
1       nick    24
2       doping  25
3       caizhi  26
4       liaozhi 27
5       wind    30
Time taken: 1.442 seconds
15/03/04 14:42:06 INFO CliDriver: Time taken: 1.442 seconds
spark-sql> 15/03/04 14:42:06 INFO StatsReportListener: task result size:(count: 2, mean: 1835.500000, stdev: 7.500000, max: 1843.000000, min: 1828.000000)
15/03/04 14:42:06 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:42:06 INFO StatsReportListener:     1828.0 B        1828.0 B        1828.0 B        1828.0 B        1843.0 B1843.0 B        1843.0 B        1843.0 B        1843.0 B
15/03/04 14:42:06 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 63 ms on localhost (2/2)
15/03/04 14:42:06 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/03/04 14:42:06 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 73.640093, stdev: 5.386124, max: 79.026217, min: 68.253968)
15/03/04 14:42:06 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:42:06 INFO StatsReportListener:     68 %    68 %    68 %    68 %    79 %    79 %    79 %    79 %    79 %
15/03/04 14:42:06 INFO StatsReportListener: other time pct: (count: 2, mean: 26.359907, stdev: 5.386124, max: 31.746032, min: 20.973783)
15/03/04 14:42:06 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:42:06 INFO StatsReportListener:     21 %    21 %    21 %    21 %    32 %    32 %    32 %    32 %    32 %

JOIN操作
select a.*,b.* from student a  join course b where a.id=b.id ;

select a.*,b.* from student a  join course b where a.id=b.id ;
15/03/04 14:45:18 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/04 14:45:18 INFO ParseDriver: Parsing command: select a.*,b.* from student a  join course b where a.id=b.id
15/03/04 14:45:18 INFO ParseDriver: Parse Completed
15/03/04 14:45:18 INFO HiveMetaStore: 0: get_table : db=default tbl=student
15/03/04 14:45:18 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_table : db=default tbl=student
15/03/04 14:45:18 INFO HiveMetaStore: 0: get_table : db=default tbl=course
15/03/04 14:45:18 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_table : db=default tbl=course
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(444571) called with curMem=1014900, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 434.2 KB, free 265.9 MB)
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(47070) called with curMem=1459471, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 46.0 KB, free 265.8 MB)
15/03/04 14:45:19 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:39062 (size: 46.0 KB, free: 267.1 MB)
15/03/04 14:45:19 INFO BlockManagerMaster: Updated info of block broadcast_8_piece0
15/03/04 14:45:19 INFO SparkContext: Created broadcast 8 from broadcast at TableReader.scala:68
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(444627) called with curMem=1506541, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 434.2 KB, free 265.4 MB)
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(47091) called with curMem=1951168, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 46.0 KB, free 265.4 MB)
15/03/04 14:45:19 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:39062 (size: 46.0 KB, free: 267.1 MB)
15/03/04 14:45:19 INFO BlockManagerMaster: Updated info of block broadcast_9_piece0
15/03/04 14:45:19 INFO SparkContext: Created broadcast 9 from broadcast at TableReader.scala:68
15/03/04 14:45:19 INFO FileInputFormat: Total input paths to process : 1
15/03/04 14:45:19 INFO SparkContext: Starting job: collect at BroadcastHashJoin.scala:53
15/03/04 14:45:19 INFO DAGScheduler: Got job 4 (collect at BroadcastHashJoin.scala:53) with 2 output partitions (allowLocal=false)
15/03/04 14:45:19 INFO DAGScheduler: Final stage: Stage 4(collect at BroadcastHashJoin.scala:53)
15/03/04 14:45:19 INFO DAGScheduler: Parents of final stage: List()
15/03/04 14:45:19 INFO DAGScheduler: Missing parents: List()
15/03/04 14:45:19 INFO DAGScheduler: Submitting Stage 4 (MappedRDD[23] at map at BroadcastHashJoin.scala:53), which has no missing parents
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(6552) called with curMem=1998259, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 6.4 KB, free 265.4 MB)
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(3857) called with curMem=2004811, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 3.8 KB, free 265.4 MB)
15/03/04 14:45:19 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:39062 (size: 3.8 KB, free: 267.1 MB)
15/03/04 14:45:19 INFO BlockManagerMaster: Updated info of block broadcast_10_piece0
15/03/04 14:45:19 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:838
15/03/04 14:45:19 INFO DAGScheduler: Submitting 2 missing tasks from Stage 4 (MappedRDD[23] at map at BroadcastHashJoin.scala:53)
15/03/04 14:45:19 INFO TaskSchedulerImpl: Adding task set 4.0 with 2 tasks
15/03/04 14:45:19 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 7, localhost, ANY, 1320 bytes)
15/03/04 14:45:19 INFO Executor: Running task 0.0 in stage 4.0 (TID 7)
15/03/04 14:45:19 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/course/course.txt:0+60
15/03/04 14:45:19 WARN LazyStruct: Extra bytes detected at the end of the row! Ignoring similar problems.
15/03/04 14:45:19 INFO Executor: Finished task 0.0 in stage 4.0 (TID 7). 2091 bytes result sent to driver
15/03/04 14:45:19 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 8, localhost, ANY, 1320 bytes)
15/03/04 14:45:19 INFO Executor: Running task 1.0 in stage 4.0 (TID 8)
15/03/04 14:45:19 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/course/course.txt:60+61
15/03/04 14:45:19 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 7) in 26 ms on localhost (1/2)
15/03/04 14:45:19 INFO Executor: Finished task 1.0 in stage 4.0 (TID 8). 2044 bytes result sent to driver
15/03/04 14:45:19 INFO DAGScheduler: Stage 4 (collect at BroadcastHashJoin.scala:53) finished in 0.042 s
15/03/04 14:45:19 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@40fa4929
15/03/04 14:45:19 INFO StatsReportListener: task runtime:(count: 2, mean: 25.500000, stdev: 0.500000, max: 26.000000, min: 25.000000)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     25.0 ms 25.0 ms 25.0 ms 25.0 ms 26.0 ms 26.0 ms 26.0 ms 26.0 ms 26.0 ms
15/03/04 14:45:19 INFO DAGScheduler: Job 4 finished: collect at BroadcastHashJoin.scala:53, took 0.075328 s
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(2264) called with curMem=2008668, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 2.2 KB, free 265.3 MB)
15/03/04 14:45:19 INFO StatsReportListener: task result size:(count: 2, mean: 2067.500000, stdev: 23.500000, max: 2091.000000, min: 2044.000000)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     2044.0 B        2044.0 B        2044.0 B        2044.0 B        2.0 KB  2.0 KB   2.0 KB  2.0 KB  2.0 KB
15/03/04 14:45:19 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 8) in 25 ms on localhost (2/2)
15/03/04 14:45:19 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
15/03/04 14:45:19 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 51.153846, stdev: 8.846154, max: 60.000000, min: 42.307692)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     42 %    42 %    42 %    42 %    60 %    60 %    60 %    60 %    60 %
15/03/04 14:45:19 INFO StatsReportListener: other time pct: (count: 2, mean: 48.846154, stdev: 8.846154, max: 57.692308, min: 40.000000)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     40 %    40 %    40 %    40 %    58 %    58 %    58 %    58 %    58 %
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(350) called with curMem=2010932, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 350.0 B, free 265.3 MB)
15/03/04 14:45:19 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:39062 (size: 350.0 B, free: 267.1 MB)
15/03/04 14:45:19 INFO BlockManagerMaster: Updated info of block broadcast_11_piece0
15/03/04 14:45:19 INFO SparkContext: Created broadcast 11 from broadcast at BroadcastHashJoin.scala:55
15/03/04 14:45:19 INFO FileInputFormat: Total input paths to process : 1
15/03/04 14:45:19 INFO SparkContext: Starting job: collect at SparkPlan.scala:84
15/03/04 14:45:19 INFO DAGScheduler: Got job 5 (collect at SparkPlan.scala:84) with 2 output partitions (allowLocal=false)
15/03/04 14:45:19 INFO DAGScheduler: Final stage: Stage 5(collect at SparkPlan.scala:84)
15/03/04 14:45:19 INFO DAGScheduler: Parents of final stage: List()
15/03/04 14:45:19 INFO DAGScheduler: Missing parents: List()
15/03/04 14:45:19 INFO DAGScheduler: Submitting Stage 5 (MappedRDD[29] at map at SparkPlan.scala:84), which has no missing parents
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(11896) called with curMem=2011282, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_12 stored as values in memory (estimated size 11.6 KB, free 265.3 MB)
15/03/04 14:45:19 INFO MemoryStore: ensureFreeSpace(6623) called with curMem=2023178, maxMem=280248975
15/03/04 14:45:19 INFO MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 6.5 KB, free 265.3 MB)
15/03/04 14:45:19 INFO BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:39062 (size: 6.5 KB, free: 267.1 MB)
15/03/04 14:45:19 INFO BlockManagerMaster: Updated info of block broadcast_12_piece0
15/03/04 14:45:19 INFO SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:838
15/03/04 14:45:19 INFO DAGScheduler: Submitting 2 missing tasks from Stage 5 (MappedRDD[29] at map at SparkPlan.scala:84)
15/03/04 14:45:19 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks
15/03/04 14:45:19 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 9, localhost, ANY, 1318 bytes)
15/03/04 14:45:19 INFO Executor: Running task 0.0 in stage 5.0 (TID 9)
15/03/04 14:45:19 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/student/stu.txt:0+28
15/03/04 14:45:19 INFO Executor: Finished task 0.0 in stage 5.0 (TID 9). 1930 bytes result sent to driver
15/03/04 14:45:19 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 10, localhost, ANY, 1318 bytes)
15/03/04 14:45:19 INFO Executor: Running task 1.0 in stage 5.0 (TID 10)
15/03/04 14:45:19 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 9) in 27 ms on localhost (1/2)
15/03/04 14:45:19 INFO HadoopRDD: Input split: hdfs://feng01:9000/user/hive/warehouse/student/stu.txt:28+29
15/03/04 14:45:19 INFO Executor: Finished task 1.0 in stage 5.0 (TID 10). 1884 bytes result sent to driver
15/03/04 14:45:19 INFO DAGScheduler: Stage 5 (collect at SparkPlan.scala:84) finished in 0.066 s
15/03/04 14:45:19 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@4942722a
15/03/04 14:45:19 INFO DAGScheduler: Job 5 finished: collect at SparkPlan.scala:84, took 0.094749 s
1       nick    24      1       英语    中文    法文    日文
2       doping  25      2       中文    法文
3       caizhi  26      3       中文    法文    日文
4       liaozhi 27      4       中文    法文    拉丁
5       wind    30      5       中文    法文    德文
Time taken: 0.612 seconds
15/03/04 14:45:19 INFO CliDriver: Time taken: 0.612 seconds
spark-sql> 15/03/04 14:45:19 INFO StatsReportListener: task runtime:(count: 2, mean: 34.000000, stdev: 7.000000, max: 41.000000, min: 27.000000)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     27.0 ms 27.0 ms 27.0 ms 27.0 ms 41.0 ms 41.0 ms 41.0 ms 41.0 ms 41.0 ms
15/03/04 14:45:19 INFO StatsReportListener: task result size:(count: 2, mean: 1907.000000, stdev: 23.000000, max: 1930.000000, min: 1884.000000)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     1884.0 B        1884.0 B        1884.0 B        1884.0 B        1930.0 B1930.0 B        1930.0 B        1930.0 B        1930.0 B
15/03/04 14:45:19 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 71.725384, stdev: 8.762421, max: 80.487805, min: 62.962963)
15/03/04 14:45:19 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 10) in 41 ms on localhost (2/2)
15/03/04 14:45:19 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     63 %    63 %    63 %    63 %    80 %    80 %    80 %    80 %    80 %
15/03/04 14:45:19 INFO StatsReportListener: other time pct: (count: 2, mean: 28.274616, stdev: 8.762421, max: 37.037037, min: 19.512195)
15/03/04 14:45:19 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 14:45:19 INFO StatsReportListener:     20 %    20 %    20 %    20 %    37 %    37 %    37 %    37 %    37 %


5：集群运行

是CLI启动一个SparkSQL应用程序的参数，如果不设置--master的话，将在启动spark-sql的机器以local方式运行，只能通过http://机器名:4040进行监控

./bin/spark-sql --master spark://feng02:7077 --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar

[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql --master spark://feng02:7077 --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar
Spark assembly has been built with Hive, including Datanucleus jars on classpath
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/03/04 15:36:50 INFO SecurityManager: Changing view acls to: jifeng
15/03/04 15:36:50 INFO SecurityManager: Changing modify acls to: jifeng
15/03/04 15:36:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jifeng); users with modify permissions: Set(jifeng)
15/03/04 15:36:50 INFO Slf4jLogger: Slf4jLogger started
15/03/04 15:36:50 INFO Remoting: Starting remoting
15/03/04 15:36:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@feng02:51996]
15/03/04 15:36:51 INFO Utils: Successfully started service 'sparkDriver' on port 51996.
15/03/04 15:36:51 INFO SparkEnv: Registering MapOutputTracker
15/03/04 15:36:51 INFO SparkEnv: Registering BlockManagerMaster
15/03/04 15:36:51 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20150304153651-d089
15/03/04 15:36:51 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
15/03/04 15:36:51 INFO HttpFileServer: HTTP File server directory is /tmp/spark-f3705b9a-d878-441f-8128-042e0363b2d3
15/03/04 15:36:51 INFO HttpServer: Starting HTTP Server
15/03/04 15:36:51 INFO Utils: Successfully started service 'HTTP file server' on port 47583.
15/03/04 15:36:51 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/03/04 15:36:51 INFO SparkUI: Started SparkUI at http://feng02:4040
15/03/04 15:36:51 INFO AppClient$ClientActor: Connecting to master spark://feng02:7077...
15/03/04 15:36:52 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20150304153652-0001
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor added: app-20150304153652-0001/0 on worker-20150304143413-feng03-41090 (feng03:41090) with 4 cores
15/03/04 15:36:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150304153652-0001/0 on hostPort feng03:41090 with 4 cores, 512.0 MB RAM
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor added: app-20150304153652-0001/1 on worker-20150304143413-feng01-51840 (feng01:51840) with 4 cores
15/03/04 15:36:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150304153652-0001/1 on hostPort feng01:51840 with 4 cores, 512.0 MB RAM
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor added: app-20150304153652-0001/2 on worker-20150304142824-feng02-56719 (feng02:56719) with 1 cores
15/03/04 15:36:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20150304153652-0001/2 on hostPort feng02:56719 with 1 cores, 512.0 MB RAM
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/1 is now LOADING
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/0 is now LOADING
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/0 is now RUNNING
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/2 is now LOADING
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/1 is now RUNNING
15/03/04 15:36:52 INFO AppClient$ClientActor: Executor updated: app-20150304153652-0001/2 is now RUNNING
15/03/04 15:36:52 INFO NettyBlockTransferService: Server created on 44473
15/03/04 15:36:52 INFO BlockManagerMaster: Trying to register BlockManager
15/03/04 15:36:52 INFO BlockManagerMasterActor: Registering block manager feng02:44473 with 267.3 MB RAM, BlockManagerId(<driver>, feng02, 44473)
15/03/04 15:36:52 INFO BlockManagerMaster: Registered BlockManager
15/03/04 15:36:53 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
SET spark.sql.hive.version=0.13.1
15/03/04 15:36:55 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@feng03:41011/user/Executor#987795855] with ID 0
15/03/04 15:36:55 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@feng01:34997/user/Executor#-664106945] with ID 1
15/03/04 15:36:55 INFO HiveMetaStore: 0: get_all_databases
15/03/04 15:36:55 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_all_databases
15/03/04 15:36:55 INFO BlockManagerMasterActor: Registering block manager feng03:42326 with 265.4 MB RAM, BlockManagerId(0, feng03, 42326)
15/03/04 15:36:55 INFO BlockManagerMasterActor: Registering block manager feng01:60774 with 265.4 MB RAM, BlockManagerId(1, feng01, 60774)
15/03/04 15:36:55 INFO HiveMetaStore: 0: get_functions: db=default pat=*
15/03/04 15:36:55 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_functions: db=default pat=*
15/03/04 15:36:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
spark-sql> 15/03/04 15:36:58 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@feng02:45328/user/Executor#-2134631037] with ID 2
15/03/04 15:36:58 INFO BlockManagerMasterActor: Registering block manager feng02:60870 with 267.3 MB RAM, BlockManagerId(2, feng02, 60870)

         >

在集群监控页面可以看到启动了SparkSQL应用程序：


数据量少的时候慢好多

> select a.*,b.* from student a  join course b where a.id=b.id ;
15/03/04 15:44:53 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/04 15:44:53 INFO ParseDriver: Parsing command: select a.*,b.* from student a  join course b where a.id=b.id
15/03/04 15:44:53 INFO ParseDriver: Parse Completed
15/03/04 15:44:55 INFO HiveMetaStore: 0: get_table : db=default tbl=student
15/03/04 15:44:55 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_table : db=default tbl=student
15/03/04 15:44:55 INFO HiveMetaStore: 0: get_table : db=default tbl=course
15/03/04 15:44:55 INFO audit: ugi=jifeng        ip=unknown-ip-addr      cmd=get_table : db=default tbl=course
15/03/04 15:44:55 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/03/04 15:44:56 INFO MemoryStore: ensureFreeSpace(436469) called with curMem=0, maxMem=280248975
15/03/04 15:44:56 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 426.2 KB, free 266.9 MB)
15/03/04 15:44:56 INFO MemoryStore: ensureFreeSpace(46914) called with curMem=436469, maxMem=280248975
15/03/04 15:44:56 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 45.8 KB, free 266.8 MB)
15/03/04 15:44:56 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on feng02:44473 (size: 45.8 KB, free: 267.2 MB)
15/03/04 15:44:56 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
15/03/04 15:44:56 INFO SparkContext: Created broadcast 0 from broadcast at TableReader.scala:68
15/03/04 15:44:56 INFO MemoryStore: ensureFreeSpace(436525) called with curMem=483383, maxMem=280248975
15/03/04 15:44:56 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 426.3 KB, free 266.4 MB)
15/03/04 15:44:56 INFO MemoryStore: ensureFreeSpace(46946) called with curMem=919908, maxMem=280248975
15/03/04 15:44:56 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 45.8 KB, free 266.3 MB)
15/03/04 15:44:56 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on feng02:44473 (size: 45.8 KB, free: 267.2 MB)
15/03/04 15:44:56 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/03/04 15:44:56 INFO SparkContext: Created broadcast 1 from broadcast at TableReader.scala:68
15/03/04 15:44:57 WARN HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead
15/03/04 15:44:58 INFO FileInputFormat: Total input paths to process : 1
15/03/04 15:44:58 INFO SparkContext: Starting job: collect at BroadcastHashJoin.scala:53
15/03/04 15:44:58 INFO DAGScheduler: Got job 0 (collect at BroadcastHashJoin.scala:53) with 2 output partitions (allowLocal=false)
15/03/04 15:44:58 INFO DAGScheduler: Final stage: Stage 0(collect at BroadcastHashJoin.scala:53)
15/03/04 15:44:58 INFO DAGScheduler: Parents of final stage: List()
15/03/04 15:44:58 INFO DAGScheduler: Missing parents: List()
15/03/04 15:44:58 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[4] at map at BroadcastHashJoin.scala:53), which has no missing parents
15/03/04 15:44:58 INFO MemoryStore: ensureFreeSpace(6544) called with curMem=966854, maxMem=280248975
15/03/04 15:44:58 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 6.4 KB, free 266.3 MB)
15/03/04 15:44:58 INFO MemoryStore: ensureFreeSpace(3848) called with curMem=973398, maxMem=280248975
15/03/04 15:44:58 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.8 KB, free 266.3 MB)
15/03/04 15:44:58 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on feng02:44473 (size: 3.8 KB, free: 267.2 MB)
15/03/04 15:44:58 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/03/04 15:44:58 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
15/03/04 15:44:58 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MappedRDD[4] at map at BroadcastHashJoin.scala:53)
15/03/04 15:44:58 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
15/03/04 15:44:58 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, feng02, NODE_LOCAL, 1320 bytes)
15/03/04 15:44:59 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on feng02:60870 (size: 3.8 KB, free: 267.3 MB)
15/03/04 15:45:01 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, feng01, ANY, 1320 bytes)
15/03/04 15:45:02 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on feng01:60774 (size: 3.8 KB, free: 265.4 MB)
15/03/04 15:45:02 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on feng02:60870 (size: 45.8 KB, free: 267.2 MB)
15/03/04 15:45:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on feng01:60774 (size: 45.8 KB, free: 265.4 MB)
15/03/04 15:45:06 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 4445 ms on feng01 (1/2)
15/03/04 15:45:07 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 9432 ms on feng02 (2/2)
15/03/04 15:45:07 INFO DAGScheduler: Stage 0 (collect at BroadcastHashJoin.scala:53) finished in 9.436 s
15/03/04 15:45:07 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
15/03/04 15:45:07 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@4ba4074f
15/03/04 15:45:07 INFO DAGScheduler: Job 0 finished: collect at BroadcastHashJoin.scala:53, took 9.687128 s
15/03/04 15:45:08 INFO MemoryStore: ensureFreeSpace(2264) called with curMem=977246, maxMem=280248975
15/03/04 15:45:08 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.2 KB, free 266.3 MB)
15/03/04 15:45:08 INFO MemoryStore: ensureFreeSpace(350) called with curMem=979510, maxMem=280248975
15/03/04 15:45:08 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 350.0 B, free 266.3 MB)
15/03/04 15:45:08 INFO BlockManager: Removing broadcast 2
15/03/04 15:45:08 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on feng02:44473 (size: 350.0 B, free: 267.2 MB)
15/03/04 15:45:08 INFO BlockManager: Removing block broadcast_2
15/03/04 15:45:08 INFO BlockManagerMaster: Updated info of block broadcast_3_piece0
15/03/04 15:45:08 INFO SparkContext: Created broadcast 3 from broadcast at BroadcastHashJoin.scala:55
15/03/04 15:45:08 INFO MemoryStore: Block broadcast_2 of size 6544 dropped from memory (free 279275659)
15/03/04 15:45:08 INFO BlockManager: Removing block broadcast_2_piece0
15/03/04 15:45:08 INFO MemoryStore: Block broadcast_2_piece0 of size 3848 dropped from memory (free 279279507)
15/03/04 15:45:08 INFO StatsReportListener: task runtime:(count: 2, mean: 6938.500000, stdev: 2493.500000, max: 9432.000000, min: 4445.000000)
15/03/04 15:45:08 INFO BlockManagerInfo: Removed broadcast_2_piece0 on feng02:44473 in memory (size: 3.8 KB, free: 267.2 MB)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/03/04 15:45:08 INFO BlockManagerInfo: Removed broadcast_2_piece0 on feng01:60774 in memory (size: 3.8 KB, free: 265.4 MB)
15/03/04 15:45:08 INFO StatsReportListener:     4.4 s   4.4 s   4.4 s   4.4 s   9.4 s   9.4 s   9.4 s   9.4 s   9.4 s
15/03/04 15:45:08 INFO StatsReportListener: fetch wait time:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:     0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms  0.0 ms
15/03/04 15:45:08 INFO StatsReportListener: remote bytes read:(count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:     0.0 B   0.0 B   0.0 B   0.0 B   0.0 B   0.0 B   0.0 B   0.0 B   0.0 B
15/03/04 15:45:08 INFO BlockManagerInfo: Removed broadcast_2_piece0 on feng02:60870 in memory (size: 3.8 KB, free: 267.2 MB)
15/03/04 15:45:08 INFO ContextCleaner: Cleaned broadcast 2
15/03/04 15:45:08 INFO StatsReportListener: task result size:(count: 2, mean: 2146.000000, stdev: 105.000000, max: 2251.000000, min: 2041.000000)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:     2041.0 B        2041.0 B        2041.0 B        2041.0 B        2.2 KB  2.2 KB   2.2 KB  2.2 KB  2.2 KB
15/03/04 15:45:08 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 96.539434, stdev: 1.668793, max: 98.208227, min: 94.870641)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:     95 %    95 %    95 %    95 %    98 %    98 %    98 %    98 %    98 %
15/03/04 15:45:08 INFO StatsReportListener: fetch wait time pct: (count: 1, mean: 0.000000, stdev: 0.000000, max: 0.000000, min: 0.000000)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:      0 %     0 %     0 %     0 %     0 %     0 %     0 %     0 %     0 %
15/03/04 15:45:08 INFO StatsReportListener: other time pct: (count: 2, mean: 3.460566, stdev: 1.668793, max: 5.129359, min: 1.791773)
15/03/04 15:45:08 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:08 INFO StatsReportListener:      2 %     2 %     2 %     2 %     5 %     5 %     5 %     5 %     5 %
15/03/04 15:45:09 INFO FileInputFormat: Total input paths to process : 1
15/03/04 15:45:09 INFO SparkContext: Starting job: collect at SparkPlan.scala:84
15/03/04 15:45:09 INFO DAGScheduler: Got job 1 (collect at SparkPlan.scala:84) with 2 output partitions (allowLocal=false)
15/03/04 15:45:09 INFO DAGScheduler: Final stage: Stage 1(collect at SparkPlan.scala:84)
15/03/04 15:45:09 INFO DAGScheduler: Parents of final stage: List()
15/03/04 15:45:09 INFO DAGScheduler: Missing parents: List()
15/03/04 15:45:09 INFO DAGScheduler: Submitting Stage 1 (MappedRDD[10] at map at SparkPlan.scala:84), which has no missing parents
15/03/04 15:45:09 INFO MemoryStore: ensureFreeSpace(11896) called with curMem=969468, maxMem=280248975
15/03/04 15:45:09 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 11.6 KB, free 266.3 MB)
15/03/04 15:45:09 INFO MemoryStore: ensureFreeSpace(6628) called with curMem=981364, maxMem=280248975
15/03/04 15:45:09 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 6.5 KB, free 266.3 MB)
15/03/04 15:45:09 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on feng02:44473 (size: 6.5 KB, free: 267.2 MB)
15/03/04 15:45:09 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
15/03/04 15:45:09 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:838
15/03/04 15:45:09 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MappedRDD[10] at map at SparkPlan.scala:84)
15/03/04 15:45:09 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
15/03/04 15:45:09 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, feng02, NODE_LOCAL, 1318 bytes)
15/03/04 15:45:09 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on feng02:60870 (size: 6.5 KB, free: 267.2 MB)
15/03/04 15:45:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on feng02:60870 (size: 45.8 KB, free: 267.2 MB)
15/03/04 15:45:09 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on feng02:60870 (size: 350.0 B, free: 267.2 MB)
15/03/04 15:45:09 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, feng02, NODE_LOCAL, 1318 bytes)
15/03/04 15:45:09 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 788 ms on feng02 (1/2)
15/03/04 15:45:09 INFO DAGScheduler: Stage 1 (collect at SparkPlan.scala:84) finished in 0.847 s
15/03/04 15:45:09 INFO StatsReportListener: Finished stage: org.apache.spark.scheduler.StageInfo@4379057
15/03/04 15:45:09 INFO StatsReportListener: task runtime:(count: 2, mean: 432.000000, stdev: 356.000000, max: 788.000000, min: 76.000000)
15/03/04 15:45:09 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:09 INFO StatsReportListener:     76.0 ms 76.0 ms 76.0 ms 76.0 ms 788.0 ms        788.0 ms        788.0 ms788.0 ms        788.0 ms
15/03/04 15:45:09 INFO DAGScheduler: Job 1 finished: collect at SparkPlan.scala:84, took 0.898729 s
15/03/04 15:45:09 INFO StatsReportListener: task result size:(count: 2, mean: 1904.000000, stdev: 23.000000, max: 1927.000000, min: 1881.000000)
15/03/04 15:45:09 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:09 INFO StatsReportListener:     1881.0 B        1881.0 B        1881.0 B        1881.0 B        1927.0 B1927.0 B        1927.0 B        1927.0 B        1927.0 B
15/03/04 15:45:09 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 76 ms on feng02 (2/2)
15/03/04 15:45:09 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/03/04 15:45:09 INFO StatsReportListener: executor (non-fetch) time pct: (count: 2, mean: 79.889126, stdev: 15.415442, max: 95.304569, min: 64.473684)
15/03/04 15:45:09 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:09 INFO StatsReportListener:     64 %    64 %    64 %    64 %    95 %    95 %    95 %    95 %    95 %
15/03/04 15:45:09 INFO StatsReportListener: other time pct: (count: 2, mean: 20.110874, stdev: 15.415442, max: 35.526316, min: 4.695431)
15/03/04 15:45:09 INFO StatsReportListener:     0%      5%      10%     25%     50%     75%     90%     95%     100%
15/03/04 15:45:09 INFO StatsReportListener:      5 %     5 %     5 %     5 %    36 %    36 %    36 %    36 %    36 %
1       nick    24      1       英语    中文    法文    日文
2       doping  25      2       中文    法文
3       caizhi  26      3       中文    法文    日文
4       liaozhi 27      4       中文    法文    拉丁
5       wind    30      5       中文    法文    德文
Time taken: 16.941 seconds
15/03/04 15:45:10 INFO CliDriver: Time taken: 16.941 seconds
---------------------
作者：wind520
来源：优快云
原文：https://blog.youkuaiyun.com/wind520/article/details/44059709
版权声明：本文为博主原创文章，转载请附上博文链接！

【大数据入门核心技术-Spark】（二十一）failed to load class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
06-10 368
摘要：运行spark-sql时出现加载Hive类失败的错误，提示需要包含-Phive和-Phive-thriftserver模块。该问题是由于安装的Spark版本为不含Hadoop库的"headless"版本，缺少Spark hive和hive thriftserver包。解决方法是从Spark官网下载包含Hadoop的预编译版本，如spark-3.1.3-bin-hadoop3.2.tgz，替换当前安装版本即可解决兼容性问题。
Spark SQL性能优化高频面试题及答案
qq_42568323的博客
09-18 1028
数据倾斜是指在Join或GroupBy操作中，某些分区的数据量远大于其他分区，导致部分任务处理时间过长，影响整体性能。优化方法：使用salting给出现数据倾斜的字段添加随机数，使其数据更加均匀分布。通过或coalesce合理调整分区数，确保任务负载均衡。对于小表和大表Join时，使用广播Join避免数据倾斜。
ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver解决办法
feinifi的博客
04-09 5921
我们通过源码编译的spark-2.3.0来启动spark-sql进行sql操作，结果出现如下错误：Spark assembly has been built with Hive, including Datanucleus jars on classpath   java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftse...
Spark问题集
CD
12-20 4393
1、java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.()V 在练习spark的过程中，有的同学遇到了这样的问题： ? 1 2 3 4 5 6 7 8 9 10 11 12
Linux 学习hadoop、spark过程中的问题记录
weixin_44769880的博客
05-22 940
1. 安装教程（分布式、伪分布式） hadoop集群安装部署——参考链接 2. 学习过程的一些记录 tar 压缩、解压缩 压缩 tar -zcvf renwolesshel.tar.gz /renwolesshel 解压缩 tar -zxvf renwolesshel.tar.gz tar命令解压缩——参考链接 3.问题记录 3.1配置阶段 3.1.1 问题：jps发现没有secondary namenode: 查看日志信息 可以看到是因为端口已经被占用了，联想到刚开始配置hdfs-site.
Error: Failed to load class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
只愿你日后不会觉得遗憾
03-25 6146
项目场景： 项目场景：hive与SparkSQL整合时，出现该报错，严格意义上来说是 使用SparkSQL命令时遇到该报错 问题描述 在使用SparkSQL命令时遇到该报错 java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lan
Failed to load org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver: org/apache/hadoop/hive/cli/
liyaya0201的博客
04-03 6361
1. 环境介绍 集群：CDH5.12.0 Spark版本：2.4.0（未编译） 执行 spark-sql 的 shell 脚本时，会报如下错误 [root@master bin]# sh spark-sql Warning: Failed to load org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver: org/apache/had...
精选资源
Spark 学习之路，包含 Spark Core，Spark SQL，Spark Streaming，Spark mllib 学
05-15
Spark 学习之路，包含 Spark Core，Spark SQL，Spark Streaming，Spark mllib 学习笔记 * [spark core学习笔记及代码 * [spark sql学习笔记及代码 * [spark streaming学习笔记及代码 Spark 消息通信 ### Spark ...
用spark连接sql数据库，报错，如何解决？
**My Coding Family**
07-29 1541
🏆本文收录于《优快云问答解惑-专业版》专栏，主要记录项目实战过程中的Bug之前因后果及提供真实有效的解决方案，希望能够助你一臂之力，帮你早日登顶实现财富自由🚀；同时，欢迎大家关注&&收藏&&订阅！持续更新中，up！up！up！！
Spark面试必问：Spark sql优化最全总结(持续更新）
大模型大数据攻城狮的专栏
11-01 810
Spark SQL是Apache Spark生态系统中的核心模块，专门用于处理。作为Spark的重要组成部分，它不仅提供了强大的SQL查询功能，还实现了与Spark其他模块的无缝集成。Spark SQL的一个关键特点是其，能够轻松地从各种数据源（如Hive、RDD、Parquet文件等）获取数据，并支持多种性能优化技术，如内存列存储和字节码生成。此外，Spark SQL引入了DataFrame和DataSet这两个高级抽象，极大地简化了开发过程，提高了开发效率。
【互联网大厂实习经验】Spark SQL数据倾斜调优圣经（含实战分享）
不积跬步，无以至千里
08-29 2104
在滴滴实习期间接触到的调优实战，先整理如下，希望可以相互学习。
Spark SQL CLI 运行
热门推荐
风行天下
03-04 1万+
1：运行 ./bin/spark-sql 需要先把hive-site.xml 负责到spark的conf目录下 [jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql Spark assembly has been built with Hive, including Datanucleus jars on classpath java.l
spark-sql 启动报错 Failed to load class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
未知的博客
04-07 296
包含hadoop、hive等相关依赖。：最简打包，不强绑定其他依赖。的spark版本就能解决该报错。运行spark-sql报错。有两个版本的spark。以3.3.4版本为例。
SPARK java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
80后大叔爱学习
03-15 1281
解决：编译时候缺少了相关jar包，需要重新编译：添加内容-Phive-thriftserver ./dev/make-distribution.sh \ --name "hadoop3-without-hive" \ --tgz "-Pyarn -Phive-thriftserver,hadoop-3.1,scala-2.11,parquet-provided,orc-provided" \ -Dhadoop.version=3.1.3 \ -Dscala.version=2.11.8 \ -Dsca..
Apache Spark启动spark-sql报错
终回首的博客
08-02 3059
一、问题 安装好spark后，执行spark-sql报错Exception in thread “main” java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT 命令 ./bin/spark-sql 报错日志： 2021-08-02 15:00:04,213 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin
安装Spark3之后Spark-SQL读不到之前安装好的Hive 数据库
qq_39841823的博客
01-14 1271
安装Spark3之后Spark-SQL读不到之前安装好的Hive 数据库前言一、问题？二、解决方案方法1.引入hive-site.xml方法2.很多步骤 ><1.增加CDH环境配置2.增加spark配置2.1 spark-defaults.conf2.2 spark-env.sh2.3 log4j.properties2.4 分发 spark软件包2.5 执行spark-sql 前言 因为之前已经 安装过CDH 其中版本为 hive 2.1.1与spark 2.2.0 , 再另外安装Spa..
Error: Failed to load class demo1 提交Spark作业报错
Xuejiujiu的博客
05-15 2707
出现在我们打包的jar包中，是以demo2命名的，所以提交demo会提示没有找到class。
spark-submit 任务提交指定类名错误解决：Error: Failed to load class
xiaoxue2333的博客
11-30 3687
在提交spark任务的时候，若 --class参数类名指定错误会让任务无法运行。找到自己需要运行的任务，就可以直接使用spark-submit命令上传任务了。那么如果不会看文件路径的话，如何精准找出自己打包的类名呢？
HiBench执行spark sql
最新发布
09-03
<think> 嗯，用户需要了解HiBench执行Spark SQL的具体方法和步骤。从用户提供的引用材料来看，HiBench确实支持Spark SQL测试，引用[3]提到它包含SQL测试项，引用[4]更是直接提到"HiBench测试Spark SQL"并给出了任务示例。用户显然是在大数据性能测试场景下工作的工程师或研究人员。 用户可能正在准备Spark集群的性能评估，需要可操作的指导。深层需求可能是想确保测试过程规范且结果可比对，因此除了基础步骤外，测试报告解读和参数配置的注意事项也很重要。 整理执行流程时要注意几个关键点：首先是配置文件调整，引用[1]提到的"hibench.scale.profile"设置很重要；其次是环境验证，用户需要确认测试环境和依赖；最后是报告生成位置，引用[4]显示任务会生成特定格式的日志。 考虑到用户的技术背景，回复要包含可粘贴的命令行操作，但也要提示潜在坑点：比如引用[4]显示prepare阶段会启动MAPREDUCE任务，而run阶段才是Spark任务，这个执行差异需要明确说明。配置项补充上，除了基础路径设置，还应该提醒用户注意Executor内存等运行时参数优化。 报告部分特别强调位置/usr/local/hibench/report，因为所有引用都未明确说明报告路径，这是结合Hibench默认安装路径补充的关键信息。最后补充性能对比建议，因为用户提供的引用[2][3]都提到Hibench的核心价值是跨框架性能评估。 </think> 以下是使用 HiBench 执行 Spark SQL 测试的标准方法和步骤，基于引用内容整理并补充关键细节： ### 🔧 一、环境准备 1. **安装配置 HiBench** - 下载 HiBench 源码：`git clone https://github.com/Intel-bigdata/HiBench.git` - 配置环境变量（`conf/hibench.conf`）： ```bash # 设置测试规模（myscale 需自定义） hibench.scale.profile = myscale # 指定 Spark 路径 spark.home = /path/to/spark # 指定 Hadoop 路径 hadoop.home = /path/to/hadoop ``` ### 🚀 二、执行 Spark SQL 测试 #### 步骤 1：初始化测试数据 ```bash # 进入 SQL 测试目录 cd bin/workloads/sql/aggregation # 示例：聚合测试（可选 join/scan 等） # 生成输入数据（触发 MapReduce 任务） ./prepare/prepare.sh ``` > 📌 **注意**： > - `prepare.sh` 会启动 **MapReduce 任务**生成 HDFS 测试数据（如引用4）[^4] > - 数据规模由 `hibench.scale.profile` 控制（引用1）[^1] #### 步骤 2：运行 Spark SQL 任务 ```bash # 执行测试（启动 Spark 作业） ./spark/run.sh ``` > 📌 **关键过程**： > - `run.sh` 提交 **Spark 任务**到集群（如引用4的 `application_xxxx`）[^4] > - 任务对应 SQL 操作：聚合（aggregation）、连接（join）等 #### 步骤 3：查看测试报告 报告路径：`/usr/local/hibench/report/hibench.report` 包含指标： - **执行时间**（秒） - **吞吐量**（MB/s） - **资源利用率**（CPU/内存，引用2）[^2] ### ️ 三、注意事项 1. **配置校验** - 确认 `hibench.conf` 中 `spark.master` 是否正确（如 `yarn` 或 `local[*]`） - 检查 HDFS 路径权限：`hibench.hdfs.data.dir` 需可读写 2. **SQL 测试类型** HiBench 支持多种 SQL 场景（引用3）[^3]： ```bash bin/workloads/sql/aggregation # 聚合 bin/workloads/sql/join # 连接 bin/workloads/sql/scan # 扫描 ``` 3. **规模调整建议** - 首次测试建议用 `tiny` 配置：`hibench.scale.profile = tiny` - 生产测试可逐步增大至 `large`（需集群资源匹配） 4. **错误排查** - 日志路径：`/usr/local/hibench/hibench/logs/` - 若 `prepare.sh` 失败，检查 Hadoop 集群状态及 HDFS 空间 ### 📊 示例输出解析 ```bash # 报告片段示例 SQL[aggregation] runtime: 125s Throughput: 32.5 MB/s Resource: CPU avg=85%, MEM avg=76% ``` > 表明：Spark SQL 聚合任务耗时 125 秒，吞吐量 32.5MB/s，资源占用较高[^2][^3]。 --- ### ❓相关问题 1. 如何自定义 HiBench 中的 Spark SQL 查询语句？ 2. HiBench 测试 Spark SQL 时如何监控集群资源消耗？ 3. 对比 HiBench 中的 Spark SQL 与 Hive on Spark 性能差异的方法？ 4. 如何扩展 HiBench 支持自定义 Spark SQL 基准测试场景？ [^1]: 规模配置与任务启动方式 [^2]: 测试指标定义（吞吐量/资源利用率） [^3]: SQL 测试类型与框架支持 [^4]: 任务执行流程（prepare/run 阶段）
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值