装载CSV数据源、装载json数据源、spark读取jar包执行scala操作

装载CSV数据源

sparkcontext操作

val conf = new SparkConf().setMaster("local[*]").setAppName("csvDemo")
    val sc = SparkContext.getOrCreate(conf)
    val lines = sc.textFile("in/users.csv")
    /*加载CSV文件，去除head
    val line2 = lines.mapPartitionsWithIndex((index, value) => {
      if (index == 0)
        value.drop(1)
      else
        value
    })
    val line3 = line2.map(x=>x.split(","))
    for (x<- line3){
      println(x.toList)
    }*/
    /* 加载CSV 去除第一行
    val line1 = lines.filter(x=>x.startsWith("user_id")==false).map(x=>x.split(","))
    line1.collect().foreach(x=>println(x.toList))*/

sparksession

val sprk = SparkSession.builder().appName("sparksession").master("locla[*]").getOrCreate()
    val df = sprk.read.format("csv").option("header", true)
      .load("in/users.csv")
   df.printSchema()
    df.show(10)
//    df.select("user_id","birthyear").show(10)
//    df.select("user_id","birthyear").printSchema()
    //修改birthyear的schma的String类型为Double
    val df2 = df.select("user_id","birthyear")
    val df3 = df2.withColumn("birthyear", df2("birthyear").cast(DoubleType))
    df3.printSchema()
    df3.filter(x => !x.isNullAt(1) && x.getDouble(1) > 1995).show(10)
    df2.withColumnRenamed("birthyear","birthYear")

装载json数据源

sparkcontext操作

    val conf = new SparkConf().setMaster("local[*]").setAppName("json")
    val sc = SparkContext.getOrCreate(conf)
    val lines: RDD[String] = sc.textFile("in/user.json")
    import scala.util.parsing.json.JSON
    val rdd = lines.map(x=>JSON.parseFull(x))
    rdd.collect().foreach(println)

sparksession操作

    val spark = SparkSession.builder().master("local[*]").appName("jsonsession").getOrCreate()
    val frame = spark.read.format("json").option("header",true).load("in/user.json")
    frame.printSchema()
    frame.show()

spark读取jar包执行scala操作

配置文件：

scala代码：

   val conf = new SparkConf().setMaster("local[*]").setAppName("json")
    val sc = SparkContext.getOrCreate(conf)
    val properties = new Properties()
    //test.properties存放在Linux上的路径
    properties.load(new FileInputStream("/root/test.properties"))
    val path: String = properties.getProperty("path")
    val savepath: String = properties.getProperty("savepath")
    val rdd: RDD[String] = sc.textFile(path)
    val result: RDD[(String, Int)] = rdd.flatMap(x => x.split(" ")).map(x => (x, 1)).reduceByKey(_ + _)
    result.foreach(println)
    result.saveAsTextFile(savepath)

打包：

ProjectStructure-Artifacts-"+"-JARS-from modules with dependencies-Main_class（类）、module(工程名)-Apply-OK
Build-Build Artifact-Build

删除jar包中的安全文件

1、在上传Linux前，手动删除jar包中的安全文件
META-INF文件夹下的 .DSA和.SF文件
2、Linux上删除

	zip -d /opt/kb09File/sparkdemo1.jar 'META-INF/*.DSA' 'META-INF/*SF'

读取包执行scala操作

spark-submit --class nj.kb11.HelloWorld --master local[*] ./sparkdemo.jar