idea利用sbt打包scala程序为jar包并发布到集群中测试

最新推荐文章于 2025-05-26 12:10:34 发布

原创最新推荐文章于 2025-05-26 12:10:34 发布 · 1.1w 阅读

25 ·

CC 4.0 BY-SA版权

欢迎转载，注明作者和出处就好！如果有任何问题或文章存在明显的谬误，请留言说明原因谢谢，我也可以知道原因，不断进步！

文章标签：

#scala #compile #package #jar #sbt

【大数据】➣ Spark 同时被 2 个专栏收录

17 篇文章

订阅专栏

【编程语言】➣ Scala

6 篇文章

订阅专栏

前提条件是创建好了wordcount项目，可以参考Scala官方IDE教程

这里写图片描述
我们需要做的是打包项目为jar包并发布到集群中去运行，下面逐一讲解打包步骤。
第一步 打开项目结构配置页面可以使用快捷键 Ctrl+Alt+Shift+S

第二步 添加jar包配置

第三步 去除额外的lib包依赖，不将其他依赖打包到jar文件中，只保留class编译文件及META-INF文件夹
这里写图片描述

第四步 编译构建生成jar包

这里写图片描述
● jar包解压后内部结构

第五步最后将得到的jar包，可上传到集群中执行

sftp> put wordcount.jar
Uploading wordcount.jar to /home/elon/workspace/wordcount/wordcount.jar
  100% 5KB      5KB/s 00:00:00     
C:/Users/yilon/Documents/wordcount.jar: 5452 bytes transferred in 0 seconds (5452 bytes/s)

[elon@hadoop spark]$ ./bin/spark-submit --class WordCount --master local ~/workspace/wordcount/wordcount.jar 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/02/22 00:38:59 INFO SparkContext: Running Spark version 2.2.1
18/02/22 00:39:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/22 00:39:01 INFO SparkContext: Submitted application: WordCount
18/02/22 00:39:01 INFO SecurityManager: Changing view acls to: elon
18/02/22 00:39:01 INFO SecurityManager: Changing modify acls to: elon
18/02/22 00:39:01 INFO SecurityManager: Changing view acls groups to: 
18/02/22 00:39:01 INFO SecurityManager: Changing modify acls groups to: 
18/02/22 00:39:01 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(elon); groups with view permissions: Set(); users  with modify permissions: Set(elon); groups with modify permissions: Set()
18/02/22 00:39:02 INFO Utils: Successfully started service 'sparkDriver' on port 44048.
18/02/22 00:39:02 INFO SparkEnv: Registering MapOutputTracker
18/02/22 00:39:02 INFO SparkEnv: Registering BlockManagerMaster
18/02/22 00:39:02 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/02/22 00:39:02 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/02/22 00:39:02 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-df8c9e80-53ba-42e5-98f9-6010211bac1c
18/02/22 00:39:02 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
18/02/22 00:39:03 INFO SparkEnv: Registering OutputCommitCoordinator
18/02/22 00:39:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/02/22 00:39:03 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.115:4040
18/02/22 00:39:04 INFO SparkContext: Added JAR file:/home/elon/workspace/wordcount/wordcount.jar at spark://192.168.1.115:44048/jars/wordcount.jar with timestamp 1519231144007
18/02/22 00:39:04 INFO Executor: Starting executor ID driver on host localhost
18/02/22 00:39:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32785.
18/02/22 00:39:04 INFO NettyBlockTransferService: Server created on 192.168.1.115:32785
18/02/22 00:39:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/02/22 00:39:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.115, 32785, None)
18/02/22 00:39:04 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.115:32785 with 413.9 MB RAM, BlockManagerId(driver, 192.168.1.115, 32785, None)
18/02/22 00:39:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.115, 32785, None)
18/02/22 00:39:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.115, 32785, None)
wordCounts: 
(package,1)
(For,3)
(Programs,1)
(processing.,1)
(Because,1)
(The,1)
(page](http://spark.apache.org/documentation.html).,1)
(cluster.,1)
(its,1)
([run,1)
(than,1)
(APIs,1)
(have,1)
(Try,1)
(computation,1)
(through,1)
(several,1)
(This,2)
(graph,1)
(Hive,2)
(storage,1)
(["Specifying,1)
(To,2)
("yarn",1)
(Once,1)
(["Useful,1)
(prefer,1)
(SparkPi,2)
(engine,1)
(version,1)
(file,1)
(documentation,,1)
(processing,,1)
(the,24)
(are,1)
(systems.,1)
(params,1)
(not,1)
(different,1)
(refer,2)
(Interactive,2)
(R,,1)
(given.,1)
(if,4)
(build,4)
(when,1)
(be,2)
(Tests,1)
(Apache,1)
(thread,1)
(programs,,1)
(including,4)
(./bin/run-example,2)
(Spark.,1)
(package.,1)
(1000).count(),1)
(Versions,1)
(HDFS,1)
(Data.,1)
(>>>,1)
(Maven,1)
(programming,1)
(Testing,1)
(module,,1)
(Streaming,1)
(environment,1)
(run:,1)
(Developer,1)
(clean,1)
(1000:,2)
(rich,1)
(GraphX,1)
(Please,4)
(is,6)
(guide](http://spark.apache.org/contributing.html),1)
(run,7)
(URL,,1)
(threads.,1)
(same,1)
(MASTER=spark://host:7077,1)
(on,7)
(built,1)
(against,1)
([Apache,1)
(tests,2)
(examples,2)
(at,2)
(optimized,1)
(3"](https://cwiki.apache.org/confluence/display/MAVEN/Parallel+builds+in+Maven+3).,1)
(usage,1)
(development,1)
(Maven,,1)
(graphs,1)
(talk,1)
(Shell,2)
(class,2)
(abbreviated,1)
(using,5)
(directory.,1)
(README,1)
(computing,1)
(overview,1)
(`examples`,2)
(example:,1)
(##,9)
(N,1)
(set,2)
(use,3)
(Hadoop-supported,1)
(running,1)
(find,1)
(contains,1)
(project,1)
(Pi,1)
(need,1)
(or,3)
(Big,1)
(high-level,1)
(Java,,1)
(uses,1)
(<class>,1)
(Hadoop,,2)
(available,1)
(requires,1)
((You,1)
(more,1)
(see,3)
(Documentation,1)
(of,5)
(tools,1)
(using:,1)
(cluster,2)
(must,1)
(supports,2)
(built,,1)
(tests](http://spark.apache.org/developer-tools.html#individual-tests).,1)
(system,1)
(build/mvn,1)
(Hadoop,3)
(this,1)
(Version"](http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version),1)
(particular,2)
(Python,2)
(Spark,16)
(general,3)
(YARN,,1)
(pre-built,1)
([Configuration,1)
(locally,2)
(library,1)
(A,1)
(locally.,1)
(sc.parallelize(1,1)
(only,1)
(Configuration,1)
(following,2)
(basic,1)
(#,1)
(changed,1)
(More,1)
(which,2)
(learning,,1)
(first,1)
(./bin/pyspark,1)
(also,4)
(info,1)
(should,2)
(for,12)
([params]`.,1)
(documentation,3)
([project,1)
(mesos://,1)
(Maven](http://maven.apache.org/).,1)
(setup,1)
(<http://spark.apache.org/>,1)
(latest,1)
(your,1)
(MASTER,1)
(example,3)
(["Parallel,1)
(scala>,1)
(DataFrames,,1)
(provides,1)
(configure,1)
(distributions.,1)
(can,7)
(About,1)
(instructions.,1)
(do,2)
(easiest,1)
(no,1)
(project.,1)
(how,3)
(`./bin/run-example,1)
(started,1)
(Note,1)
(by,1)
(individual,1)
(spark://,1)
(It,2)
(tips,,1)
(Scala,2)
(Alternatively,,1)
(an,4)
(variable,1)
(submit,1)
(-T,1)
(machine,1)
(thread,,1)
(them,,1)
(detailed,2)
(stream,1)
(And,1)
(distribution,1)
(review,1)
(return,2)
(Thriftserver,1)
(developing,1)
(./bin/spark-shell,1)
("local",1)
(start,1)
(You,4)
(Spark](#building-spark).,1)
(one,3)
(help,1)
(with,4)
(print,1)
(Spark"](http://spark.apache.org/docs/latest/building-spark.html).,1)
(data,1)
(Contributing,1)
(in,6)
(-DskipTests,1)
(downloaded,1)
(versions,1)
(online,1)
(Guide](http://spark.apache.org/docs/latest/configuration.html),1)
(builds,1)
(comes,1)
(Tools"](http://spark.apache.org/developer-tools.html).,1)
([building,1)
(Python,,2)
(Many,1)
(building,2)
(Running,1)
(from,1)
(way,1)
(Online,1)
(site,,1)
(other,1)
(Example,1)
([Contribution,1)
(analysis.,1)
(sc.parallelize(range(1000)).count(),1)
(you,4)
(runs.,1)
(Building,1)
(higher-level,1)
(protocols,1)
(guidance,2)
(a,8)
(guide,,1)
(name,1)
(fast,1)
(SQL,2)
(that,2)
(will,1)
(IDE,,1)
(to,17)
(get,1)
(,71)
(information,1)
(core,1)
(web,1)
("local[N]",1)
(programs,2)
(option,1)
(MLlib,1)
(["Building,1)
(contributing,1)
(shell:,2)
(instance:,1)
(Scala,,1)
(and,9)
(command,,2)
(package.),1)
(./dev/run-tests,1)
(sample,1)