insert overwrite table case_data_sample select * from case_data_sample_tmp;
2025-06-18 16:37:06,500 INFO [main] conf.HiveConf: Using the default value passed in for log id: 531f6207-2ea7-471a-9eac-9ce1e6a79910
2025-06-18 16:37:06,500 INFO [main] session.SessionState: Updating thread name to 531f6207-2ea7-471a-9eac-9ce1e6a79910 main
2025-06-18 16:37:06,503 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Compiling command(queryId=root_20250618163706_29623b7c-a221-4e37-9b54-e05709d7f990): insert overwrite table case_data_sample select * from case_data_sample_tmp
2025-06-18 16:37:06,546 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Concurrency mode is disabled, not creating a lock manager
2025-06-18 16:37:06,547 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Starting Semantic Analysis
2025-06-18 16:37:06,559 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Completed phase 1 of Semantic Analysis
2025-06-18 16:37:06,560 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for source tables
2025-06-18 16:37:06,588 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for subqueries
2025-06-18 16:37:06,588 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for destination tables
2025-06-18 16:37:06,627 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Completed getting MetaData in Semantic Analysis
2025-06-18 16:37:08,746 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for source tables
2025-06-18 16:37:08,784 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for subqueries
2025-06-18 16:37:08,784 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for destination tables
2025-06-18 16:37:08,884 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] common.FileUtils: Creating directory if it doesn't exist: hdfs://master:8020/user/hive/warehouse/ad_traffic.db/case_data_sample/.hive-staging_hive_2025-06-18_16-37-06_538_2993870488593298816-1
2025-06-18 16:37:09,012 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Generate an operator pipeline to autogather column stats for table ad_traffic.case_data_sample in query insert overwrite table case_data_sample select * from case_data_sample_tmp
2025-06-18 16:37:09,069 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for source tables
2025-06-18 16:37:09,098 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for subqueries
2025-06-18 16:37:09,098 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Get metadata for destination tables
2025-06-18 16:37:09,155 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Context: New scratch dir is hdfs://master:8020/user/hive/tmp/root/531f6207-2ea7-471a-9eac-9ce1e6a79910/hive_2025-06-18_16-37-09_012_5684077500801740374-1
2025-06-18 16:37:09,221 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] common.FileUtils: Creating directory if it doesn't exist: hdfs://master:8020/user/hive/tmp/root/531f6207-2ea7-471a-9eac-9ce1e6a79910/hive_2025-06-18_16-37-09_012_5684077500801740374-1/-mr-10000/.hive-staging_hive_2025-06-18_16-37-09_012_5684077500801740374-1
2025-06-18 16:37:09,234 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: CBO Succeeded; optimized logical plan.
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for FS(2)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for FS(9)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for SEL(8)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for GBY(7)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for RS(6)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for GBY(5)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for SEL(4)
2025-06-18 16:37:09,329 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for SEL(1)
2025-06-18 16:37:09,330 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ppd.OpProcFactory: Processing for TS(0)
2025-06-18 16:37:09,385 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] optimizer.ColumnPrunerProcFactory: RS 6 oldColExprMap: {VALUE._col20=Column[_col20], VALUE._col10=Column[_col10], VALUE._col21=Column[_col21], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2025-06-18 16:37:09,386 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] optimizer.ColumnPrunerProcFactory: RS 6 newColExprMap: {VALUE._col20=Column[_col20], VALUE._col10=Column[_col10], VALUE._col21=Column[_col21], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2025-06-18 16:37:09,500 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SetSparkReducerParallelism: Number of reducers for sink RS[6] was already determined to be: 1
2025-06-18 16:37:09,646 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Examining input format to see if vectorization is enabled.
2025-06-18 16:37:09,655 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Vectorization is enabled for input format(s) [org.apache.hadoop.mapred.TextInputFormat]
2025-06-18 16:37:09,655 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Validating and vectorizing MapWork... (vectorizedVertexNum 0)
2025-06-18 16:37:09,706 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map vectorization enabled: true
2025-06-18 16:37:09,706 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map vectorized: false
2025-06-18 16:37:09,707 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map notVectorizedReason: Aggregation Function expression for GROUPBY operator: UDF compute_stats not supported
2025-06-18 16:37:09,707 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map vectorizedVertexNum: 0
2025-06-18 16:37:09,707 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map enabledConditionsMet: [hive.vectorized.use.vector.serde.deserialize IS true]
2025-06-18 16:37:09,707 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Map inputFileFormatClassNameSet: [org.apache.hadoop.mapred.TextInputFormat]
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Validating and vectorizing ReduceWork... (vectorizedVertexNum 1)
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reduce vectorization enabled: true
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reduce vectorized: false
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reduce notVectorizedReason: Aggregation Function expression for GROUPBY operator: UDF compute_stats not supported
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reduce vectorizedVertexNum: 1
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reducer hive.vectorized.execution.reduce.enabled: true
2025-06-18 16:37:09,753 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] physical.Vectorizer: Reducer engine: spark
2025-06-18 16:37:09,784 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] parse.CalcitePlanner: Completed plan generation
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Semantic Analysis Completed (retrial = false)
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: , FieldSchema(name:case_data_sample_tmp.timestamps, type:int, comment:null), FieldSchema(name:case_data_sample_tmp.camp, type:int, comment:null), FieldSchema(name:case_data_sample_tmp.creativeid, type:int, comment:null), FieldSchema(name:case_data_sample_tmp.mobile_os, type:int, comment:null), FieldSchema(name:case_data_sample_tmp.mobile_type, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.app_key_md5, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.app_name_md5, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.placementid, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.useragent, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.mediaid, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.os_type, type:string, comment:null), FieldSchema(name:case_data_sample_tmp.born_time, type:int, comment:null), FieldSchema(name:case_data_sample_tmp.label, type:int, comment:null)], properties:null)
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Completed compiling command(queryId=root_20250618163706_29623b7c-a221-4e37-9b54-e05709d7f990); Time taken: 3.282 seconds
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] reexec.ReExecDriver: Execution #1 of query
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Concurrency mode is disabled, not creating a lock manager
2025-06-18 16:37:09,785 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Executing command(queryId=root_20250618163706_29623b7c-a221-4e37-9b54-e05709d7f990): insert overwrite table case_data_sample select * from case_data_sample_tmp
2025-06-18 16:37:09,786 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Query ID = root_20250618163706_29623b7c-a221-4e37-9b54-e05709d7f990
2025-06-18 16:37:09,786 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Total jobs = 1
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Launching Job 1 out of 1
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: In order to change the average load for a reducer (in bytes):
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: set hive.exec.reducers.bytes.per.reducer=<number>
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: In order to limit the maximum number of reducers:
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: set hive.exec.reducers.max=<number>
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: In order to set a constant number of reducers:
2025-06-18 16:37:09,813 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: set mapreduce.job.reduces=<number>
2025-06-18 16:37:09,834 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] session.SparkSessionManagerImpl: Setting up the session manager.
2025-06-18 16:37:10,327 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] session.SparkSession: Trying to open Spark session 4201eecc-977e-4e69-89ec-50403379b3d2
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 4201eecc-977e-4e69-89ec-50403379b3d2)'
2025-06-18 16:37:10,372 ERROR [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: .lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.generateSparkConf(HiveSparkClientFactory.java:263)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:98)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:76)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:87)
... 24 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:359)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more
2025-06-18 16:37:10,378 ERROR [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 4201eecc-977e-4e69-89ec-50403379b3d2)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 4201eecc-977e-4e69-89ec-50403379b3d2
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:221)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:92)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:115)
at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136)
at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:115)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.generateSparkConf(HiveSparkClientFactory.java:263)
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:98)
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:76)
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:87)
... 24 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:359)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 28 more
2025-06-18 16:37:10,391 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] reexec.ReOptimizePlugin: ReOptimization: retryPossible: false
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 4201eecc-977e-4e69-89ec-50403379b3d2
2025-06-18 16:37:10,391 ERROR [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 4201eecc-977e-4e69-89ec-50403379b3d2
2025-06-18 16:37:10,392 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Completed executing command(queryId=root_20250618163706_29623b7c-a221-4e37-9b54-e05709d7f990); Time taken: 0.607 seconds
2025-06-18 16:37:10,392 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] ql.Driver: Concurrency mode is disabled, not creating a lock manager
2025-06-18 16:37:10,433 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] conf.HiveConf: Using the default value passed in for log id: 531f6207-2ea7-471a-9eac-9ce1e6a79910
2025-06-18 16:37:10,433 INFO [531f6207-2ea7-471a-9eac-9ce1e6a79910 main] session.SessionState: Resetting thread name to main
扫一扫
444
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
