Spark上传jar包到Hadoop(Linux)

已于 2022-10-20 08:16:00 修改
阅读量1.5k 收藏 2
点赞数 1
文章标签： spark jar hadoop
于 2022-10-17 22:53:56 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/m0_63086929/article/details/127378737
版权

1.在idea中添加pom打包依赖

        <pluginManagement>
            <plugins>
                <!-- 编译scala的插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- 编译java的插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>

        如果出错可以将3.2.2换成3.2.0

2.生命周期package打成jar包

3.将jar包上传到linux中  可以利用Xshell

        rz快捷键

4.先启动Hadoop,Spark...

        start-all.sh

5.可以先打开spark的webui界面 ip:8080

        也可以通过spark-shell命令查看端口

6.利用spark-submit上传到集群

[root@hadoop ~]# spark-submit --class Task.wordCount --master local /root/spark_study-1.0.jar

这里是本地上传所以是local 也可以是 spark://ip:7077

完整提交命令：

spark-submit --class 包名.object名 --master 地址 --executor-memory  1g --total-executor-cores 4 jar包路径

7.展示

[root@hadoop ~]# cd /usr/local/src/spark
[root@hadoop spark]# spark-submit --class Task.task01.Step01 --master loca /root/spark_study-1.0.jarError: Master must either be yarn or start with spark, mesos, local
Run with --help for usage help or --verbose for debug output
[root@hadoop spark]# cd bin
[root@hadoop bin]# spark-submit --class Task.task01.Step01 --master local /root/spark_study-1.0.jar
22/10/20 07:53:09 INFO spark.SparkContext: Running Spark version 2.1.1
22/10/20 07:53:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/10/20 07:53:10 INFO spark.SecurityManager: Changing view acls to: root
22/10/20 07:53:10 INFO spark.SecurityManager: Changing modify acls to: root
22/10/20 07:53:10 INFO spark.SecurityManager: Changing view acls groups to: 
22/10/20 07:53:10 INFO spark.SecurityManager: Changing modify acls groups to: 
22/10/20 07:53:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/10/20 07:53:10 INFO util.Utils: Successfully started service 'sparkDriver' on port 37223.
22/10/20 07:53:11 INFO spark.SparkEnv: Registering MapOutputTracker
22/10/20 07:53:11 INFO spark.SparkEnv: Registering BlockManagerMaster
22/10/20 07:53:11 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/10/20 07:53:11 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/10/20 07:53:11 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-bf622dee-70e2-4e23-9002-4ced1155e705
22/10/20 07:53:11 INFO memory.MemoryStore: MemoryStore started with capacity 413.9 MB
22/10/20 07:53:11 INFO spark.SparkEnv: Registering OutputCommitCoordinator
22/10/20 07:53:11 INFO util.log: Logging initialized @2282ms
22/10/20 07:53:11 INFO server.Server: jetty-9.2.z-SNAPSHOT
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5af28b27{/jobs,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@71104a4{/jobs/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4985cbcb{/jobs/job,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@72f46e16{/jobs/job/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3c9168dc{/stages,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@332a7fce{/stages/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@549621f3{/stages/stage,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@54361a9{/stages/stage/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@32232e55{/stages/pool,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5217f3d0{/stages/pool/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@37ebc9d8{/storage,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@293bb8a5{/storage/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2416a51{/storage/rdd,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6fa590ba{/storage/rdd/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6e9319f{/environment,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@72e34f77{/environment/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7bf9b098{/executors,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@389adf1d{/executors/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77307458{/executors/threadDump,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1fc0053e{/executors/threadDump/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@290b1b2e{/static,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@47874b25{/,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@33617539{/api,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2c177f9e{/jobs/job/kill,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5db4c359{/stages/stage/kill,null,AVAILABLE,@Spark}
22/10/20 07:53:11 INFO server.ServerConnector: Started Spark@d78795{HTTP/1.1}{0.0.0.0:4040}
22/10/20 07:53:11 INFO server.Server: Started @2482ms
22/10/20 07:53:11 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
22/10/20 07:53:11 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.60.130:4040
22/10/20 07:53:11 INFO spark.SparkContext: Added JAR file:/root/spark_study-1.0.jar at spark://192.168.60.130:37223/jars/spark_study-1.0.jar with timestamp 1666223591637
22/10/20 07:53:11 INFO executor.Executor: Starting executor ID driver on host localhost
22/10/20 07:53:11 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 35308.
22/10/20 07:53:11 INFO netty.NettyBlockTransferService: Server created on 192.168.60.130:35308
22/10/20 07:53:11 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/10/20 07:53:11 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.60.130, 35308, None)
22/10/20 07:53:11 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.60.130:35308 with 413.9 MB RAM, BlockManagerId(driver, 192.168.60.130, 35308, None)
22/10/20 07:53:11 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.60.130, 35308, None)
22/10/20 07:53:11 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.60.130, 35308, None)
22/10/20 07:53:11 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@36a7abe1{/metrics/json,null,AVAILABLE,@Spark}
22/10/20 07:53:11 WARN spark.SparkContext: Using an existing SparkContext; some configuration may not take effect.
22/10/20 07:53:12 INFO internal.SharedState: Warehouse path is 'file:/usr/local/src/spark/bin/spark-warehouse'.
22/10/20 07:53:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40620d8e{/SQL,null,AVAILABLE,@Spark}
22/10/20 07:53:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@49b07ee3{/SQL/json,null,AVAILABLE,@Spark}
22/10/20 07:53:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@28486680{/SQL/execution,null,AVAILABLE,@Spark}
22/10/20 07:53:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a1e3ac1{/SQL/execution/json,null,AVAILABLE,@Spark}
22/10/20 07:53:12 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5300f14a{/static/sql,null,AVAILABLE,@Spark}
22/10/20 07:53:13 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 243.7 KB, free 413.7 MB)
22/10/20 07:53:13 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.6 KB, free 413.7 MB)
22/10/20 07:53:13 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.60.130:35308 (size: 23.6 KB, free: 413.9 MB)
22/10/20 07:53:13 INFO spark.SparkContext: Created broadcast 0 from textFile at Step01.scala:13
22/10/20 07:53:14 INFO mapred.FileInputFormat: Total input paths to process : 1
22/10/20 07:53:14 INFO spark.SparkContext: Starting job: first at Step01.scala:14
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Got job 0 (first at Step01.scala:14) with 1 output partitions
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (first at Step01.scala:14)
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Parents of final stage: List()
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Missing parents: List()
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv MapPartitionsRDD[1] at textFile at Step01.scala:13), which has no missing parents
22/10/20 07:53:15 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.3 KB, free 413.7 MB)
22/10/20 07:53:15 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1985.0 B, free 413.7 MB)
22/10/20 07:53:15 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.60.130:35308 (size: 1985.0 B, free: 413.9 MB)
22/10/20 07:53:15 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:996
22/10/20 07:53:15 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv MapPartitionsRDD[1] at textFile at Step01.scala:13)
22/10/20 07:53:15 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
22/10/20 07:53:15 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 6065 bytes)
22/10/20 07:53:15 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
22/10/20 07:53:15 INFO executor.Executor: Fetching spark://192.168.60.130:37223/jars/spark_study-1.0.jar with timestamp 1666223591637
22/10/20 07:53:15 INFO client.TransportClientFactory: Successfully created connection to /192.168.60.130:37223 after 19 ms (0 ms spent in bootstraps)
22/10/20 07:53:15 INFO util.Utils: Fetching spark://192.168.60.130:37223/jars/spark_study-1.0.jar to /tmp/spark-867e26d9-0441-4b96-a8a3-2cc311b6b80d/userFiles-a67d8a2d-c17f-4b0c-b5cd-b969dc4b41ab/fetchFileTemp4283142326901178396.tmp
22/10/20 07:53:15 INFO executor.Executor: Adding file:/tmp/spark-867e26d9-0441-4b96-a8a3-2cc311b6b80d/userFiles-a67d8a2d-c17f-4b0c-b5cd-b969dc4b41ab/spark_study-1.0.jar to class loader
22/10/20 07:53:15 INFO rdd.HadoopRDD: Input split: hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv:0+124387
22/10/20 07:53:16 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
22/10/20 07:53:16 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
22/10/20 07:53:16 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
22/10/20 07:53:16 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
22/10/20 07:53:16 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
22/10/20 07:53:16 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 1556 bytes result sent to driver
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1154 ms on localhost (executor driver) (1/1)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
22/10/20 07:53:16 INFO scheduler.DAGScheduler: ResultStage 0 (first at Step01.scala:14) finished in 1.191 s
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Job 0 finished: first at Step01.scala:14, took 1.477301 s
22/10/20 07:53:16 INFO spark.SparkContext: Starting job: count at Step01.scala:17
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Got job 1 (count at Step01.scala:17) with 1 output partitions
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (count at Step01.scala:17)
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Parents of final stage: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Missing parents: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv MapPartitionsRDD[1] at textFile at Step01.scala:13), which has no missing parents
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 3.1 KB, free 413.7 MB)
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1915.0 B, free 413.7 MB)
22/10/20 07:53:16 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.60.130:35308 (size: 1915.0 B, free: 413.9 MB)
22/10/20 07:53:16 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:996
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv MapPartitionsRDD[1] at textFile at Step01.scala:13)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, ANY, 5983 bytes)
22/10/20 07:53:16 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
22/10/20 07:53:16 INFO rdd.HadoopRDD: Input split: hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv:0+124387
22/10/20 07:53:16 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 1210 bytes result sent to driver
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 92 ms on localhost (executor driver) (1/1)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
22/10/20 07:53:16 INFO scheduler.DAGScheduler: ResultStage 1 (count at Step01.scala:17) finished in 0.093 s
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Job 1 finished: count at Step01.scala:17, took 0.113036 s
22/10/20 07:53:16 INFO spark.SparkContext: Starting job: count at Step01.scala:17
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Got job 2 (count at Step01.scala:17) with 1 output partitions
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (count at Step01.scala:17)
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Parents of final stage: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Missing parents: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[2] at filter at Step01.scala:15), which has no missing parents
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.3 KB, free 413.7 MB)
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2017.0 B, free 413.6 MB)
22/10/20 07:53:16 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.60.130:35308 (size: 2017.0 B, free: 413.9 MB)
22/10/20 07:53:16 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:996
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[2] at filter at Step01.scala:15)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, ANY, 5983 bytes)
22/10/20 07:53:16 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
22/10/20 07:53:16 INFO rdd.HadoopRDD: Input split: hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv:0+124387
22/10/20 07:53:16 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 1123 bytes result sent to driver
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 67 ms on localhost (executor driver) (1/1)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
22/10/20 07:53:16 INFO scheduler.DAGScheduler: ResultStage 2 (count at Step01.scala:17) finished in 0.074 s
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Job 2 finished: count at Step01.scala:17, took 0.090038 s
删除数据源中缺失值大于3个字段的数据的条目数为：4条
22/10/20 07:53:16 INFO spark.SparkContext: Starting job: count at Step01.scala:18
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Got job 3 (count at Step01.scala:18) with 1 output partitions
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (count at Step01.scala:18)
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Parents of final stage: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Missing parents: List()
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[2] at filter at Step01.scala:15), which has no missing parents
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.3 KB, free 413.6 MB)
22/10/20 07:53:16 INFO memory.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2017.0 B, free 413.6 MB)
22/10/20 07:53:16 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.60.130:35308 (size: 2017.0 B, free: 413.9 MB)
22/10/20 07:53:16 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:996
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[2] at filter at Step01.scala:15)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, ANY, 5983 bytes)
22/10/20 07:53:16 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 3)
22/10/20 07:53:16 INFO rdd.HadoopRDD: Input split: hdfs://192.168.60.130:9000/file3_1/accommodationdata.csv:0+124387
22/10/20 07:53:16 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 3). 1123 bytes result sent to driver
22/10/20 07:53:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 46 ms on localhost (executor driver) (1/1)
22/10/20 07:53:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
22/10/20 07:53:16 INFO scheduler.DAGScheduler: ResultStage 3 (count at Step01.scala:18) finished in 0.047 s
22/10/20 07:53:16 INFO scheduler.DAGScheduler: Job 3 finished: count at Step01.scala:18, took 0.060488 s
清洗后输出的结果文件总行数为：1043行
22/10/20 07:53:16 INFO server.ServerConnector: Stopped Spark@d78795{HTTP/1.1}{0.0.0.0:4040}
22/10/20 07:53:16 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on 192.168.60.130:35308 in memory (size: 2017.0 B, free: 413.9 MB)
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5db4c359{/stages/stage/kill,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2c177f9e{/jobs/job/kill,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@33617539{/api,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@47874b25{/,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@290b1b2e{/static,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1fc0053e{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@77307458{/executors/threadDump,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@389adf1d{/executors/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7bf9b098{/executors,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72e34f77{/environment/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6e9319f{/environment,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6fa590ba{/storage/rdd/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2416a51{/storage/rdd,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@293bb8a5{/storage/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@37ebc9d8{/storage,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5217f3d0{/stages/pool/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32232e55{/stages/pool,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@54361a9{/stages/stage/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@549621f3{/stages/stage,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@332a7fce{/stages/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3c9168dc{/stages,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@72f46e16{/jobs/job/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4985cbcb{/jobs/job,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@71104a4{/jobs/json,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5af28b27{/jobs,null,UNAVAILABLE,@Spark}
22/10/20 07:53:16 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.60.130:35308 in memory (size: 1915.0 B, free: 413.9 MB)
22/10/20 07:53:16 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.60.130:4040
22/10/20 07:53:16 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/10/20 07:53:16 INFO memory.MemoryStore: MemoryStore cleared
22/10/20 07:53:16 INFO storage.BlockManager: BlockManager stopped
22/10/20 07:53:16 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
22/10/20 07:53:16 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/10/20 07:53:16 INFO spark.SparkContext: Successfully stopped SparkContext
22/10/20 07:53:16 INFO spark.SparkContext: SparkContext already stopped.
22/10/20 07:53:16 INFO util.ShutdownHookManager: Shutdown hook called
22/10/20 07:53:16 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-867e26d9-0441-4b96-a8a3-2cc311b6b80d
[root@hadoop bin]#

8.输出看文件行数可以有这个命令：

[root@hadoop bin]# hdfs dfs -cat /accommodationoutput1/* |wc -l
1043

北海｀
关注
spark-3.5.1+Hadoop 3.4.0+Hive4.0 分布式集群 安装配置
tonyhi6的博客
06-08 3672
三 Spark 与Hive 集成。1 拷贝配置文件和Mysql 驱动。2 登录hive，创建测试表。3 启动 spark-sql。3 安装spark。
Linux执行hdfs上的jar包,使用Hadoop命令行执行jar包详解（生成jar、将文件上传到dfs、执...
weixin_33358246的博客
05-14 1314
使用Hadoop命令行执行jar包详解(生成jar、将文件上传到dfs、执使用Hadoop命令行执行jar包详解(生成jar、将文件上传到dfs、执行命令、下载dfs文件至本地)Hadoop入门配置系列博客目录一览1、Eclipse中使用Hadoop伪分布模式开发配置及简单程序示例(Linux下)2、使用Hadoop命令行执行jar包详解(生成jar、将文件上传到dfs、执行命令、下载dfs文件至...
Spark应用(app jar)发布到Hadoop集群的过程
weixin_34037515的博客
07-19 172
记录了Spark，Hadoop集群的开启，关闭，以及Spark应用提交到Hadoop集群的过程，通过web端监控运行状态。 1.绝对路径开启集群 （每次集群重启，默认配置的hadoop集群中tmp文件被清空所以需要重新format) 我的集群安装位置是/opt/hadoop下，可以根据自己路径修改。 /opt/hadoop/...
spark常见的submit参数
最新发布
小枫的博客
03-11 855
spark-submit 是在spark安装目录中bin目录下的一个shell脚本文件，用于在集群中启动应用程序
idea导入hadoop jar包
m0_59092234的博客
04-03 4506
hadoop jar包 在hadoop安装目录下，找到sharehadoop目录，搜索jar，全选，然后在安装目录新建_jar文件夹，将所有的jar包拷进去 idea添加jar包 在Project Structure下，Libraries下添加jar包所在目录即可 使用maven管理jar包更方便，添加依赖就行了 ...
用java提交一个Spark应用程序
大数据星球-浪尖
12-09 4826
第一种方式经常有人在公众号留言或者在群里问浪尖，如何使用java代码提交一个Spark应用程序。在这里浪尖介绍一种简单的方法来实现这个功能。首先用vim打开一个文件，MyLauncher.java编写代码如下：import org.apache.spark.launcher.SparkAppHandle;import org.apache.spark.launcher.SparkLauncher;
大数据开发之词频统计传参打包成jar包发送到Hadoop运行并创建可执行文件方便运行
学习笔记
12-01 614
首先需要修改代码，将setMaster(“local[*]”)删了。点击-号，这些全删了，留最后一个compile output即可。然后需要进行瘦身，把不需要的包删了，这样打包出来就比较小了。然后选择spark文件里的jars下所有的jar包。同时点击一下，我们也可以看到打包后存放的路径。其中路径得你自己集群的文件路径才能运行成功。点击Project Structure。跳出这个，点击rebuild即可。注意jar包的路径得全路径了。此时可以测试一下传参效果。这就是传入的两个参数。可以调试的时候试试。
spark-3.4.1-bin-hadoop3.tgz - Spark 3.4.1 安装包(内置了Hadoop 3)
07-19
文件名: spark-3.4.1-bin-hadoop3.tgz 这是 Apache Spark 3.4.1 版本的二进制文件，专为与 Hadoop 3 配合使用而设计。Spark 是一种快速、通用的集群计算系统，用于大规模数据处理。这个文件包含了所有必要的组件，...
spark-3.2.1-bin-hadoop2.7.tgz
02-20
总结一下，"spark-3.2.1-bin-hadoop2.7.tgz"是一个专为Linux设计的Spark版本，与Hadoop 2.7兼容，提供了高效的大数据处理能力，涵盖了核心计算、SQL查询、流处理、机器学习和图计算等多个方面。在实际应用中，开发者...
spark2.1.0-bin-hadoop2.7
07-06
1. 下载：首先，你需要下载Spark 2.1.0与Hadoop 2.7兼容的二进制包，即`spark-2.1.0-bin-hadoop2.7.tgz`。 2. 解压：在Linux服务器上，使用`tar -zxvf spark-2.1.0-bin-hadoop2.7.tgz`命令解压文件。 3. 配置环境...
java提交spark代码_如何在Java应用中提交Spark任务？
weixin_42683394的博客
02-13 754
最近看到有几个Github友关注了Streaming的监控工程——Teddy，所以思来想去还是优化下代码，不能让别人看笑话，是不。于是就想改在一下之前最丑陋的一个地方——任务提交本博客内容基于Spark2.2版本~在阅读文章并想实际操作前，请确保你有：一台配置好Spark和yarn的服务器支持正常spark-submit --master yarn xxxx的任务提交老版本老版本任务提交是采用启动...
spark 提交java_java中使用SparkLauncher提交spark应用
weixin_32254243的博客
02-21 520
将开发好的spark application(对于java/scala来说是jar)提交到spark集群执行的方式通常包括两种，一种是通常使用的spark submit脚本(spark 2.x版本是spark2-shell，可以参考https://www.jianshu.com/p/1d41174441b6)，另一种是spark官方提供的java API SparkLauncher类。下面来介绍使...
Spark写数据到Hadoop：HDFS与 Spark如何协同工作
shuimuzh123的博客
05-12 5810
Spark作为大数据计算框架，主要的优势在于数据计算，但是因为自身不具备分布式文件系统，所以往往需要结合其他的文件系统和数据源来完成工作，这其中就包括HDFS。HDFS是Hadoop的分布式文件系统，Spark写数据到Hadoop，其实就是HDFS与Spark协同工作，那么这一过程要如何实现呢？ Spark框架不管是写数据还是读取数据，基于Hadoop集群，主要靠的就是HDFS。我们可以把这个过程分为四个阶段： 将RDD写入HDFS→读取HDFS上的文件→将HDFS上的文件添加到Driver→判断HDFS上
spark源码（一）spark-submit如何将jar以及配置参数提交给spark服务器
Interest1_wyt的博客
06-12 3187
spark-submit提交任务给spark服务器
Maven打包Java版的spark程序到jar包，本地Windows测试，上传到集群中运行
Mr.Write的博客
08-22 7523
Maven打包本地spark程序到jar包，上传到集群中运行 Step1：Maven打包Jar包 win + R 打开运行窗口输入cmd打开命令行窗口，切换到程序所在目录，我的目录是 E:\code\JavaWorkspace\SparkDemo 编译程序 mvn compile 打包程序 mvn package 如果程序没有特殊情况，比如缺胳膊少腿，打包完后回看到...
spark执行优化--将依赖的jar包上传至hdfs上（spark3.1.1）
weixin_42934205的博客
10-26 3867
spark执行过程优化------将spark运行的jar包上传到HDFS上，避免在提交任务时因为spark程序包过大导致的占用运行时间和cpu资源 起因是在提交spark任务的时候，因为服务器处于离线状态，好多的依赖包无法通过网络进行下载，所以打成的spark程序包比较大，大概在200~300MB之间，每次通过spark-on-yarn提交任务时，总是有一段时间将本地的spark程序上传到hdfs，占用了太多的网络资源以及cpu。 所以，通过配置spark.yarn.archive和spark.yarn.
Java代码封装成jar包并上传到HDFS上
qq_43881783的博客
05-12 3041
Java代码封装成jar包并上传到HDFS上运行 步骤如下： 一、Java代码封装成jar包 1、点击File找到Projexts Structure…并点击 2、找到并点击Artifacts，再点击+号 3、点击jar再点击From… 4、出现如图，选好点击OK 5、选中刚才的模块并点击build,再点击Build Artifacts… 6、出现下图并点击Build 7、打包完成后你的项目会出现一个out文件夹，里面有封装好的jar包，如图： 二、将jar包上传到HDFS上并运行 1、将jar包先上
spark添加Jar包依赖到HDFS
onway_goahead的博客
07-08 3098
问题：Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka010.KafkaUtils$ 解决方案： 1.将依赖上传到HDFS 2. spark-defalults.conf 设置： spark.yarn.jars=hdfs://node1:9000/user/bd/spark/spark...
Springboot（2.0.0.RELEASE）+spark（2.1.0）框架整合到jar包成功发布（原创）！！！
weixin_33842304的博客
08-09 358
一、前言 　　首先说明一下，这个框架的整合可能对大神来说十分容易，但是对我来说十分不易，踩了不少坑。虽然整合的时间不长，但是值得来纪念下！！！我个人开发工具比较喜欢IDEA，创建的springboot的java工程再引入scala的library之后当前工程即可创建java文件又可创建scala文件，这个一个工程里可采用java+scala双语言来开发。用了这个框架搭建成功后给我们开发spark...
Spark 1.6.0 for Hadoop 2.4环境部署包下载指南
- `jars`：存放了运行Spark应用需要的所有依赖jar包。 最后，对于想要在生产环境中部署和使用Spark 1.6.0的朋友来说，建议关注以下几个方面的内容： - 集群管理：了解如何在YARN或Mesos集群上部署Spark。 - 性能...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值