Spark dataframe【KV格式】模拟实现Map操作

代码实现　

// rdd转化为df[kv格式]
val df = sqlContext.createDataFrame(check_data_type, structType)
.select("cons_no", "org_no")
.distinct // 去重
.rdd
.map(row => (row.getAs[String]("cons_no"), row.getAs[String]("org_no")))
.collect()// 数据量较少
.toMap[String, String]
// 模式匹配
val value = df.get("2019") match{
  case Some(a) => a // 提取出数据
  case None => ""  
}
println(value)

 

转载于:https://www.cnblogs.com/yszd/p/10638738.html