kopunk的专栏

1.有header的txt文件创建DataFrame：利用 mapPartitionsWithIndexval teacherRdd = sc.textFile("src/test/teacher.txt")val teacherRddSchema = teacherRdd.mapPartitionsWithIndex((idx, iter) => if (idx == 0) i...

读取CSV文件创建dataFrame的写法：val df1 = spark.read .format("csv") .option("header","true") .option("inferSchema","true") .load("src/test/resource/1.csv")完整写法：import org.apache....

先定义一个case clascase class people(id:Int, name:String, age:Int)读取txt，转化为RDDval rddpeople = sc.textFile("source path")利用case class给RDD一个schemaval peopleSchema = rddpeople.map(row => row.s...

先创建一个rank spec// 用df("salary").asc代表升序排列，desc代表降序val rankSpec = Window.partitionBy("age").orderBy(df("salary").desc)然后为自己的表添加一列rankval sararyRank = df.withColumn("rank",dense_rank().over(ra...