sbt打包WordCount的过程

最新推荐文章于 2025-03-21 18:58:29 发布

原创

最新推荐文章于 2025-03-21 18:58:29 发布 · 1.2k 阅读

1 ·

CC 4.0 BY-SA版权

本文详述了使用sbt在Spark环境下打包WordCount应用的步骤，包括安装sbt、创建项目文件、编写Scala代码、打包、上传数据到HDFS、发布并运行jar包。最终在Spark集群上成功运行并展示了结果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.在安装有spark的客户端上安装sbt（请参考上一篇博客）

2.在一台slave终端机的/home/hadoop1/xuguokun/下创建WordCount文件夹。

3.创建build.sbt文件，该文件的里面的内容如下：

name := "o2o-spark"

version := "0.1"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
  "org.scalanlp" % "chalk" % "1.3.0",
  "org.apache.spark" %% "spark-core" % "1.3.1",
  "org.apache.spark" %% "spark-mllib" % "1.3.1",
  "org.apache.spark" % "spark-streaming_2.10" % "1.3.1",
  "org.apache.spark" %% "spark-streaming-kafka" % "1.1.0",
  "org.apache.hadoop" % "hadoop-client" % "2.2.0",
  "org.apache.hadoop" % "hadoop-common" % "2.2.0",
  "org.apache.hadoop" % "hadoop-hdfs" % "2.2.0",
  "com.github.scopt" %% "scopt" % "3.3.0",
  "org.apache.spark" %% "spark-sql" % "1.5.1",
  "org.apache.spark" %% "spark-hive" % "1.5.1",
  "org.apache.hbase" % "hbase" % "0.94.18"
)

resolvers += "Akka Repository" at "http:/