Flink之Table API 例子

使用Flink实现每10秒统计一次各渠道APP启动数量,包括设置StreamExecutionEnvironment,利用Kafka消费数据,解析JSON日志,定义TimestampExtractor以支持EventTime,将数据流转换为Table并应用窗口函数进行聚合。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

//每10秒中渠道为appstore的个数
def main(args: Array[String]): Unit = {
  //sparkcontext
  val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment

  //时间特性改为eventTime
  env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)

  val myKafkaConsumer: FlinkKafkaConsumer011[String] = MyKafkaUtil.getConsumer("GMALL_STARTUP")
  val dstream: DataStream[String] = env.addSource(myKafkaConsumer)

  val startupLogDstream: DataStream[StartupLog] = dstream.map{ jsonString =>JSON.parseObject(jsonString,classOf[StartupLog]) }
  //告知watermark 和 eventTime如何提取
  val startupLogWithEventTimeDStream: DataStream[StartupLog] = startupLogDstream.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor[StartupLog](Time.seconds(0L)) {
    override def extractTimestamp(element: StartupLog): Long = {
      element.ts
    }
  }).setParallelism(1)

  //SparkSession
  val tableEnv: StreamTableEnvironment = TableEnvironment.getTableEnvironment(env)

  //把数据流转化成Table
  val startupTable: Table = tableEnv.fromDataStream(startupLogWithEventTimeDStream , 'mid,'uid,'appid,'area,'os,'ch,'logType,'vs,'logDate,'logHour,'logHourMinute,'ts.rowtime)

  //通过table api 进行操作
  // 每10秒 统计一次各个渠道的个数 table api 解决
  //1 groupby  2 要用 window   3 用eventtime来确定开窗时间
  val resultTable: Table = startupTable.window(Tumble over 10000.millis on 'ts as 'tt).groupBy('ch,'tt ).select( 'ch, 'ch.count)
 
 

  //把Table转化成数据流
  //val appstoreDStream: DataStream[(String, String, Long)] = appstoreTable.toAppendStream[(String,String,Long)]
  val resultDstream: DataStream[(Boolean, (String, Long))] = resultSQLTable.toRetractStream[(String,Long)]

  resultDstream.filter(_._1).print()

  env.execute()

}
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值