Spark编程经典综合案例_spark案例-优快云博客

1. 求TOP值

两个文件包含4个字段orderid、userid、payment、productid，求TopN个payment值。

file1.txt

1,1768,50,155
2,1218,600,211
3,2239,788,242
4,3101,28,599
5,4899,290,129
6,3110,54,1201
7,4436,259,877
8,2369,7890,27

file2.txt

100,4287,226,233
101,6562,489,124
102,1124,33,17
103,3267,159,179
104,4569,57,125
105,1438,37,116

import org.apache.spark.sql.SparkSession

object Test {
    def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName("wc")
          .master("local")
          .getOrCreate()
        spark.sparkContext.setLogLevel("ERROR")
        val input_path = "/Users/zz/Desktop/input/"
        // 分区数目为2，2个文件
        val lines = spark.sparkContext.textFile(input_path, 2)
        var num = 0
        val result = lines.filter(line => (line.trim().length > 0)
          && (line.split(",").length == 4))
          .map(_.split(",")(2))      // 取第三列
          .map(x => (x.toInt, ""))  // 转换为kv
          .sortByKey(false)
          .map(x => x._1).take(5)
          .foreach(x => {
              num = num + 1
              println(num + "\t" + x)
          })
        spark.stop()
    }
}

输出

1   7890
2   788
3   600
4   489
5   290

2. 求最大值最小值

file1.txt

129
54
167
324
111
54
26
697
4856
3418

file2.txt

5
329
14
4567
2186
457
35
267

spark.sparkContext.setLogLevel("ERROR")
val input_path = "/Users/zenmen/Desktop/input/"
// 2为分区数目
val lines = spark.sparkContext.textFile(input_path, 2)
val max = lines.filter(line => (line.trim().length > 0))
  .map(x => (x.trim().toInt, ""))
  .sortByKey(false)    //生序则可取出最小值
  .map(x => x._1)
  .take(1)
max.foreach(println)

或

val input_path = "/Users/zenmen/Desktop/input/"
val lines = spark.sparkContext.textFile(input_path, 2)
val result = lines.filter(_.trim().length>0)
  .map(line => ("key", line.trim.toInt))
  .groupByKey()
  .map(x => {
    var min = Integer.MAX_VALUE
    var max = Integer.MIN_VALUE
    for(num <- x._2){     // 对值列表进行遍历
      if(num>max){max=num}
      if(num<min){min=num}
    }
    (max,min)
  }).collect
  .foreach(x => {
    println("max= "+x._1)
    println("min= "+x._2)
  })

输出

max= 4856
min= 5

3. 文件排序

对多个输入文件进行排序，输出文件包括rank和value两列

file1.txt

33
37
12
40

file2.txt

4
16
39
5

file3.txt

1
45
25

import org.apache.spark.HashPartitioner
val input_path = "/Users/zenmen/Desktop/input/"
val output_path = "/Users/zenmen/Desktop/output/"
// 分区数目为3
val lines = spark.sparkContext.textFile(input_path, 3)
var index = 0
val result = lines.filter(_.trim().length>0)
  .map(line => (line.trim.toInt, ""))
  .partitionBy(new HashPartitioner(1))   // 确保排序结果的准确性
  .sortByKey()
  .map(t => {
    index += 1
    (index,t._1)
  })
result.saveAsTextFile(output_path)

输出

(1,1)
(2,4)
(3,5)
(4,12)
(5,16)
(6,25)
(7,33)
(8,37)
(9,39)
(10,40)
(11,45)

4. 二次排序

对于一个输入文件，首先根据第一列进行排序，若数据相等，则根据第二列进行排序。

file.txt

5 3
1 6
4 9
8 3
5 7
5 6
3 2

实现步骤：

1. 按照Ordered 和 Serializable接口实现自定义排序的key；

2. 加载将要进行二次排序的文件，生成kv类型的RDD；

3. 使用sortByKey基于自定义的key进行二次排序；

4. 去除掉排序的key，只保留排序的结果。

class SecondarySortKey(val first:Int, val second:Int)
  extends Ordered[SecondarySortKey] with Serializable {
  override def compare(other: SecondarySortKey): Int = {
    if(this.first-other.first!=0){  // first 不相等
      this.first-other.first
    }else{
      this.second-other.second
    }
  }
}

val input_path = "/Users/zenmen/Desktop/input/file.txt"
// 分区数目为1
val lines = spark.sparkContext.textFile(input_path, 1)
val result = lines.map(line => (new SecondarySortKey(
  line.split(" ")(0).toInt,
  line.split(" ")(1).toInt),
  line)
).sortByKey(false)
.map(_._2)   // 取出line列
.collect().foreach(println)

输出：

5. 连接操作

给定两个文件，file1.txt为用户电影评分表，包含uid,movieId,score三个字段；file2.txt为电影信息表包括movieId,movieName两个字段。取出平均分大于4.0的电影。

file1.txt

1::100::3.1
2::101::6.2
3::100::9.3
4::100::3.4
5::101::7.5
6::102::6.6
7::103::2.7

file2.txt

100,huluwa
101,aoteman
102,huoying
103,sunwukong

spark.sparkContext.setLogLevel("ERROR")
val input_path_1 = "/Users/zenmen/Desktop/input/file1.txt"
val input_path_2 = "/Users/zenmen/Desktop/input/file2.txt"
val output_path = "/Users/zenmen/Desktop/output/"
val rdd1 = spark.sparkContext.textFile(input_path_1)
  .map(line => {
    val fileds = line.split("::")
    (fileds(1).toInt, fileds(2).toDouble)
  }).groupByKey()  // 电影维度进行聚合，计算得分平均值
  .map(data => {
    val avg = data._2.sum / data._2.size
    (data._1, avg)
  }
)
rdd1.foreach(println)

val rdd2 = spark.sparkContext.textFile(input_path_2)
  .map(line => {
    val fields = line.split(",")
    (fields(0).toInt, fields(1))   // 使用keyBy时，需把key转换为int
  }
).keyBy(_._1)
rdd2.foreach(println)

keyBy将第一个值作为key，将整体作为value。

val result = rdd1.keyBy(tup => tup._1).join(rdd2)  // 根据keyBy的键进行join
result.foreach(println)

val result_final = result.filter(f => f._2._1._2 > 4.0)
  .map(f => (f._1, f._2._2._2, f._2._1._2))
result_final.foreach(println)
// result_final.saveAsTextFile(output_path)