Spark存储文件，修改目录及文件名

今天业务刚好要用spark来将数据库里的数据定时存储到hdfs,分目录,改文件名,记下来防止以后不写spark忘记（老年人记性差）

第一种方式使用saveAsHadoopFile算子,继承MultipleTextOutputFormat

话不多说,上代码

//查出当天日期的前一天
	val cal = Calendar.getInstance
	cal.setTime(new Date)
	cal.add(Calendar.DATE, -1)
//存储的目录，根据自己的业务来分,存储Hdfs就写hdfs路径
   val path:String = "file:///D:/test/" + new SimpleDateFormat("yyyy").format(cal.getTime) +
     "/" + new SimpleDateFormat("MM").format(cal.getTime) +
     "/" + new SimpleDateFormat("dd").format(cal.getTime)
   val conf = new SparkConf()
   conf.setAppName("lqc")
   conf.setMaster("local[*]")
   val spark = SparkSession.builder().config(conf).getOrCreate()
   val sc: SparkContext = spark.sparkContext
//禁止生成success文件
   sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")
   val df = spark.read.jdbc(url, tableName, con)
   val value = df.selectExpr("*")
//根据create_time这个存入表的时间字段,查出上一天的数据
     .where("to_date(create_time) < to_date(now()) and to_date(create_time) > date_sub(to_date(now()),'2')")
//将df转为rdd类型的二元组，saveAsHadoopFile默认是以K,V形式存储
     .rdd.map(x => ("", x))
   value.saveAsHadoopFile(path, classOf[String], classOf[String], classOf[MyOutputFormat])

继承MultipleTextOutputFormat

class MyOutputFormat extends MultipleTextOutputFormat[Any, Any] {
  val cal = Calendar.getInstance
  cal.setTime(new Date)
  cal.add(Calendar.DATE, -1)
  override def generateFileNameForKeyValue(key: Any, value: Any, name: String): String = {
    //设置文件名字，我还是取的存入数据的前一天
    val fileName = new SimpleDateFormat("yyyyMMdd").format(cal.getTime)
    fileName
  }
  override def generateActualKey(key: Any, value: Any): String = {
    //K，不输出就好，因为V不需要更改不用重写V的方法
    ""
      }
}

保存到本地上有个crc的文件去不掉,存hdfs是没有的,回头在研究一下

第二种方式，修改文件名

        Calendar cal = Calendar.getInstance();
        cal.setTime(new Date());
        cal.add(Calendar.DATE, -1);
        //hdfs路径
        String uri = "hdfs://";
        Configuration configuration = new Configuration();
        FileSystem fs = null;
        try {
            fs = FileSystem.get(URI.create(uri), configuration);
        } catch (IOException e) {
            e.printStackTrace();
        }
        Path path = new Path("/air/" + new SimpleDateFormat("yyyy").format(cal.getTime()) + "-YEAR" +
                "/" + new SimpleDateFormat("MM").format(cal.getTime()) + "-MONTH" +
                "/" + new SimpleDateFormat("dd").format(cal.getTime()) + "-DAY");
        String file = null;
        try {
            assert fs != null;
            file = fs.globStatus(new Path(path + "/part*"))[0].getPath().getName();
        } catch (IOException e) {
            e.printStackTrace();
        }
        System.out.println(file);
        try {
            fs.rename(new Path(path + "/" + file),
                    new Path(path + "/" + new SimpleDateFormat("yyyy-MM-dd").format(cal.getTime())));
        } catch (IOException e) {
            e.printStackTrace();
        }

结果ok