Key-Value类型的RDD的创建及基本转换（1）

最新推荐文章于 2024-05-31 10:02:57 发布

天使在飞-666

最新推荐文章于 2024-05-31 10:02:57 发布

阅读量1.7k

点赞数 1

分类专栏： Spark基础编程

本文链接：https://blog.youkuaiyun.com/weixin_37614967/article/details/80498790

版权

Spark基础编程专栏收录该内容

6 篇文章

订阅专栏

1. 创建一个基本的key-value的RDD

scala> val kvPairRDD =
     |   sc.parallelize(Seq(("key1", "value1"), ("key2", "value2"), ("key3", "value3")))
kvPairRDD: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[17] at parallelize at <console>:25

//使用collect获得集群元素信息
scala> kvPairRDD.collect
res21: Array[(String, String)] = Array((key1,value1), (key2,value2), (key3,value3))

2. 可以从一个类的对象中创建RDD

case class User(userId: String, amount: Int)

val personSeqRDD =
  sc.parallelize(Seq(User("jeffy", 30), User("kkk", 20), User("jeffy", 30), User("kkk", 30)))

scala> personSeqRDD.collect
res22: Array[User] = Array(User(jeffy,30), User(kkk,20), User(jeffy,30), User(kkk,30))

scala> //将RDD变成二元组类型的RDD
scala> val keyByRDD = personSeqRDD.keyBy(x => x.userId)
keyByRDD: org.apache.spark.rdd.RDD[(String, User)] = MapPartitionsRDD[18] at keyBy at <console>:28

scala> keyByRDD.collect
res23: Array[(String, User)] = Array((jeffy,User(jeffy,30)), (kkk,User(kkk,20)), (jeffy,User(jeffy,30)), (kkk,User(kkk,30)))

val keyRDD2 = personSeqRDD.map(user => (user.userId, user))

scala> keyRDD2.collect

res24: Array[(String, User)] = Array((jeffy,User(jeffy,30)), (kkk,User(kkk,20)), (jeffy,User(jeffy,30)), (kkk,User(kkk,30)))

val keyRDD3 = personSeqRDD.map(user => (user.userId, user.amount))

scala> keyRDD3.collect
res25: Array[(String, Int)] = Array((jeffy,30), (kkk,20), (jeffy,30), (kkk,30))

val groupByRDD = personSeqRDD.groupBy(user => user.userId)

scala> val groupByRDD = personSeqRDD.groupBy(user => user.userId)

groupByRDD: org.apache.spark.rdd.RDD[(String, Iterable[User])] = ShuffledRDD[22] at groupBy at <console>:28

val rdd1 = sc.parallelize(Seq("test", "hell"))

rdd1.map(str => (str, 1))

scala> val a = rdd1.map(str => (str, 1))
a: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[25] at map at <console>:26
scala> a.collect
res27: Array[(String, Int)] = Array((test,1), (hell,1))