(/root/.conda/envs/untitled) [root@master untitled]# spark-submit /root/IdeaProjects/untitled/test.py25/
06/21
03:46:42 INFO spark.SparkContext: Running Spark version 3.2.425/
06/21
03:46:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable25/
06/21
03:46:42 INFO resource.ResourceUtils: ==============================================================25/
06/21
03:46:42 INFO resource.ResourceUtils: No custom resources configured for spark.driver.25/
06/21
03:46:42 INFO resource.ResourceUtils: ==============================================================25/
06/21
03:46:42 INFO spark.SparkContext: Submitted application: NBAPlayerStatsAnalysis25/
06/21
03:46:42 INFO resource.ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)25/
06/21
03:46:42 INFO resource.ResourceProfile: Limiting resource is cpu25/
06/21
03:46:42 INFO resource.ResourceProfileManager: Added ResourceProfile id: 025/
06/21
03:46:42 INFO spark.SecurityManager: Changing view acls to: root25/
06/21
03:46:42 INFO spark.SecurityManager: Changing modify acls to: root25/
06/21
03:46:42 INFO spark.SecurityManager: Changing view acls groups to: 25/
06/21
03:46:42 INFO spark.SecurityManager: Changing modify acls groups to: 25/
06/21
03:46:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()25/
06/21
03:46:42 INFO util.Utils: Successfully started service 'sparkDriver' on port 36933.25/
06/21
03:46:42 INFO spark.SparkEnv: Registering MapOutputTracker25/
06/21
03:46:42 INFO spark.SparkEnv: Registering BlockManagerMaster25/
06/21
03:46:42 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information25/
06/21
03:46:42 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up25/
06/21
03:46:42 INFO spark.SparkEnv: Registering BlockManagerMasterHeartbeat25/
06/21
03:46:42 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-f4bf76ca-4382-4352-8252-7091abcccdd225/
06/21
03:46:42 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MiB25/
06/21
03:46:42 INFO spark.SparkEnv: Registering OutputCommitCoordinator25/
06/21
03:46:42 INFO util.log: Logging initialized @3442ms to org.sparkproject.jetty.util.log.Slf4jLog25/
06/21
03:46:43 INFO server.Server: jetty-9.4.44.v20210927; built: 2021-09-27T23:02:44.612Z; git: 8da83308eeca865e495e53ef315a249d63ba9332; jvm 1.8.0_241-b0725/
06/21
03:46:43 INFO server.Server: Started @3522ms25/
06/21
03:46:43 INFO server.AbstractConnector: Started ServerConnector@3802b02d{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}25/
06/21
03:46:43 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@33cb530d{/jobs,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@597073aa{/jobs/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@170a0963{/jobs/job,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@d7aff6c{/jobs/job/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b3cd35f{/stages,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@749bd873{/stages/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@36f6ff64{/stages/stage,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@53c4807{/stages/stage/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3fab6100{/stages/pool,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@58d1569b{/stages/pool/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2510a308{/storage,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ae892d4{/storage/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40270ad1{/storage/rdd,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2deda930{/storage/rdd/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a1d8a12{/environment,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@16f587b6{/environment/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@216349f5{/executors,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@455d5738{/executors/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@397514a9{/executors/threadDump,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14a9355f{/executors/threadDump/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1151479d{/static,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6966d957{/,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@67db9e0c{/api,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54e22bed{/jobs/job/kill,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@16ebb574{/stages/stage/kill,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://master:404025/
06/21
03:46:43 INFO executor.Executor: Starting executor ID driver on host master25/
06/21
03:46:43 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 418
29.25/
06/21
03:46:43 INFO netty.NettyBlockTransferService: Server created on master:418
2925/
06/21
03:46:43 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy25/
06/21
03:46:43 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, master, 418
29, None)25/
06/21
03:46:43 INFO storage.BlockManagerMasterEndpoint: Registering block manager master:418
29 with 366.3 MiB RAM, BlockManagerId(driver, master, 418
29, None)25/
06/21
03:46:43 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, master, 418
29, None)25/
06/21
03:46:43 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, master, 418
29, None)25/
06/21
03:46:43 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e7cc552{/metrics/json,null,AVAILABLE,@Spark}25/
06/21
03:46:43 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.25/
06/21
03:46:43 INFO internal.SharedState: Warehouse path is 'file:/root/IdeaProjects/untitled/spark-warehouse'.25/
06/21
03:46:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4782cb47{/SQL,null,AVAILABLE,@Spark}25/
06/21
03:46:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@136cde
03{/SQL/json,null,AVAILABLE,@Spark}25/
06/21
03:46:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@212a8549{/SQL/execution,null,AVAILABLE,@Spark}25/
06/21
03:46:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54bfaa8f{/SQL/execution/json,null,AVAILABLE,@Spark}25/
06/21
03:46:44 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@
29426cde{/static/sql,null,AVAILABLE,@Spark}25/
06/21
03:46:45 INFO datasources.InMemoryFileIndex: It took 115 ms to list leaf files for 1 paths.25/
06/21
03:46:45 INFO datasources.InMemoryFileIndex: It took 3 ms to list leaf files for 1 paths.25/
06/21
03:46:47 INFO datasources.FileSourceStrategy: Pushed Filters: 25/
06/21
03:46:47 INFO datasources.FileSourceStrategy: Post-Scan Filters: (length(trim(value#0, None)) > 0)25/
06/21
03:46:47 INFO datasources.FileSourceStrategy: Output Data Schema: struct<value: string>25/
06/21
03:46:47 INFO codegen.CodeGenerator: Code generated in 205.916922 ms25/
06/21
03:46:47 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 222.8 KiB, free 366.1 MiB)25/
06/21
03:46:47 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 20.9 KiB, free 366.1 MiB)25/
06/21
03:46:47 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on master:418
29 (size: 20.9 KiB, free: 366.3 MiB)25/
06/21
03:46:47 INFO spark.SparkContext: Created broadcast 0 from csv at NativeMethodAccessorImpl.java:025/
06/21
03:46:47 INFO execution.FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.25/
06/21
03:46:48 INFO spark.SparkContext: Starting job: csv at NativeMethodAccessorImpl.java:025/
06/21
03:46:48 INFO scheduler.DAGScheduler: Got job 0 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (csv at NativeMethodAccessorImpl.java:0)25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Parents of final stage: List()25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Missing parents: List()25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 11.6 KiB, free 366.1 MiB)25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.8 KiB, free 366.0 MiB)25/
06/21
03:46:48 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on master:418
29 (size: 5.8 KiB, free: 366.3 MiB)25/
06/21
03:46:48 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:147425/
06/21
03:46:48 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))25/
06/21
03:46:48 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 025/
06/21
03:46:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (master, executor driver, partition 0, NODE_LOCAL, 4881 bytes) taskResourceAssignments Map()25/
06/21
03:46:48 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)25/
06/21
03:46:48 INFO datasources.FileScanRDD: Reading File path: hdfs://master:9000/usr/local/hadoop/clean_data_final.csv, range: 0-71444, partition values: [empty row]25/
06/21
03:46:48 INFO codegen.CodeGenerator: Code generated in 9.108527 ms25/
06/21
03:46:48 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1646 bytes result sent to driver25/
06/21
03:46:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 462 ms on master (executor driver) (1/1)25/
06/21
03:46:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 25/
06/21
03:46:48 INFO scheduler.DAGScheduler: ResultStage 0 (csv at NativeMethodAccessorImpl.java:0) finished in 0.622 s25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job25/
06/21
03:46:48 INFO scheduler.TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Job 0 finished: csv at NativeMethodAccessorImpl.java:0, took 0.659738 s25/
06/21
03:46:48 INFO codegen.CodeGenerator: Code generated in 7.9882
03 ms25/
06/21
03:46:48 INFO datasources.FileSourceStrategy: Pushed Filters: 25/
06/21
03:46:48 INFO datasources.FileSourceStrategy: Post-Scan Filters: 25/
06/21
03:46:48 INFO datasources.FileSourceStrategy: Output Data Schema: struct<value: string>25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 222.8 KiB, free 365.8 MiB)25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 20.9 KiB, free 365.8 MiB)25/
06/21
03:46:48 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on master:418
29 (size: 20.9 KiB, free: 366.3 MiB)25/
06/21
03:46:48 INFO spark.SparkContext: Created broadcast 2 from csv at NativeMethodAccessorImpl.java:025/
06/21
03:46:48 INFO execution.FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.25/
06/21
03:46:48 INFO spark.SparkContext: Starting job: csv at NativeMethodAccessorImpl.java:025/
06/21
03:46:48 INFO scheduler.DAGScheduler: Got job 1 (csv at NativeMethodAccessorImpl.java:0) with 1 output partitions25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0)25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Parents of final stage: List()25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Missing parents: List()25/
06/21
03:46:48 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at csv at NativeMethodAccessorImpl.java:0), which has no missing parents25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 16.7 KiB, free 365.8 MiB)25/
06/21
03:46:48 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.5 KiB, free 365.8 MiB)25/
06/21
03:46:48 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on master:418
29 (size: 8.5 KiB, free: 366.2 MiB)25/
06/21
03:46:48 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:147425/
06/21
03:46:48 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at csv at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0))25/
06/21
03:46:48 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 025/
06/21
03:46:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (master, executor driver, partition 0, NODE_LOCAL, 4881 bytes) taskResourceAssignments Map()25/
06/21
03:46:48 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)25/
06/21
03:46:49 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on master:418
29 in memory (size: 5.8 KiB, free: 366.3 MiB)25/
06/21
03:46:49 INFO datasources.FileScanRDD: Reading File path: hdfs://master:9000/usr/local/hadoop/clean_data_final.csv, range: 0-71444, partition values: [empty row]25/
06/21
03:46:49 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 1840 bytes result sent to driver25/
06/21
03:46:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 273 ms on master (executor driver) (1/1)25/
06/21
03:46:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 25/
06/21
03:46:49 INFO scheduler.DAGScheduler: ResultStage 1 (csv at NativeMethodAccessorImpl.java:0) finished in 0.339 s25/
06/21
03:46:49 INFO scheduler.DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job25/
06/21
03:46:49 INFO scheduler.TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished25/
06/21
03:46:49 INFO scheduler.DAGScheduler: Job 1 finished: csv at NativeMethodAccessorImpl.java:0, took 0.343278 s25/
06/21
03:46:51 INFO datasources.FileSourceStrategy: Pushed Filters: 25/
06/21
03:46:51 INFO datasources.FileSourceStrategy: Post-Scan Filters: 25/
06/21
03:46:51 INFO datasources.FileSourceStrategy: Output Data Schema: struct<pname: string, pos: string, team: string, age: int, gp: int ...
29 more fields>25/
06/21
03:46:51 WARN util.package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.25/
06/21
03:46:51 INFO codegen.CodeGenerator: Code generated in 39.64368 ms25/
06/21
03:46:51 INFO memory.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 222.6 KiB, free 365.6 MiB)25/
06/21
03:46:51 INFO memory.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 20.9 KiB, free 365.6 MiB)25/
06/21
03:46:51 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on master:418
29 (size: 20.9 KiB, free: 366.2 MiB)25/
06/21
03:46:51 INFO spark.SparkContext: Created broadcast 4 from toPandas at /root/IdeaProjects/untitled/test.py:1825/
06/21
03:46:51 INFO execution.FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.25/
06/21
03:46:51 INFO spark.SparkContext: Starting job: toPandas at /root/IdeaProjects/untitled/test.py:1825/
06/21
03:46:51 INFO scheduler.DAGScheduler: Got job 2 (toPandas at /root/IdeaProjects/untitled/test.py:18) with 1 output partitions25/
06/21
03:46:51 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (toPandas at /root/IdeaProjects/untitled/test.py:18)25/
06/21
03:46:51 INFO scheduler.DAGScheduler: Parents of final stage: List()25/
06/21
03:46:51 INFO scheduler.DAGScheduler: Missing parents: List()25/
06/21
03:46:51 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[13] at toPandas at /root/IdeaProjects/untitled/test.py:18), which has no missing parents25/
06/21
03:46:51 INFO memory.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 28.7 KiB, free 365.5 MiB)25/
06/21
03:46:51 INFO memory.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 365.5 MiB)25/
06/21
03:46:51 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on master:418
29 (size: 10.1 KiB, free: 366.2 MiB)25/
06/21
03:46:51 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:147425/
06/21
03:46:51 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[13] at toPandas at /root/IdeaProjects/untitled/test.py:18) (first 15 tasks are for partitions Vector(0))25/
06/21
03:46:51 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks resource profile 025/
06/21
03:46:51 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2) (master, executor driver, partition 0, NODE_LOCAL, 4881 bytes) taskResourceAssignments Map()25/
06/21
03:46:51 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)25/
06/21
03:46:51 INFO datasources.FileScanRDD: Reading File path: hdfs://master:9000/usr/local/hadoop/clean_data_final.csv, range: 0-71444, partition values: [empty row]25/
06/21
03:46:52 INFO codegen.CodeGenerator: Code generated in 63.838881 ms25/
06/21
03:46:52 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 94
292 bytes result sent to driver25/
06/21
03:46:52 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 130 ms on master (executor driver) (1/1)25/
06/21
03:46:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 25/
06/21
03:46:52 INFO scheduler.DAGScheduler: ResultStage 2 (toPandas at /root/IdeaProjects/untitled/test.py:18) finished in 0.137 s25/
06/21
03:46:52 INFO scheduler.DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job25/
06/21
03:46:52 INFO scheduler.TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished25/
06/21
03:46:52 INFO scheduler.DAGScheduler: Job 2 finished: toPandas at /root/IdeaProjects/untitled/test.py:18, took 0.1405
03 s25/
06/21
03:46:52 INFO spark.SparkContext: Invoking stop() from shutdown hook25/
06/21
03:46:52 INFO server.AbstractConnector: Stopped Spark@3802b02d{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}25/
06/21
03:46:52 INFO ui.SparkUI: Stopped Spark web UI at http://master:404025/
06/21
03:46:52 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!25/
06/21
03:46:52 INFO memory.MemoryStore: MemoryStore cleared25/
06/21
03:46:52 INFO storage.BlockManager: BlockManager stopped25/
06/21
03:46:52 INFO storage.BlockManagerMaster: BlockManagerMaster stopped25/
06/21
03:46:52 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!25/
06/21
03:46:52 INFO spark.SparkContext: Successfully stopped SparkContext25/
06/21
03:46:52 INFO util.ShutdownHookManager: Shutdown hook called25/
06/21
03:46:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-880bca98-c6bd-42a0-84a9-e243c4052d6f25/
06/21
03:46:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-184bfddf-35ba-4ae4-aa65-df8be18332be25/
06/21
03:46:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-184bfddf-35ba-4ae4-aa65-df8be18332be/pyspark-66ea83af-4111-4f7c-9ec6-1a6ffae82e4a
扫一扫
8882
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
