【实战】Flink实现统计每个小时的各类日志条数

最新推荐文章于 2023-11-03 10:38:57 发布

原创

最新推荐文章于 2023-11-03 10:38:57 发布 · 5.2k 阅读

19 ·

CC 4.0 BY-SA版权

文章标签：

#flink #触发器 #大数据

本文介绍了如何使用Flink进行实时数据处理，具体场景为统计每个小时各类日志的数量。通过设置合适的触发器和窗口，实现对海量日志的高效聚合分析。

Flink实现统计每个小时的各类日志条数

import java.text.SimpleDateFormat
import java.util.{Date, Properties}
import org.apache.flink.api.scala._
import com.google.gson.{JsonObject, JsonParser}
import com.iflytekzunhong.util.RedisPoolUtil
import org.apache.flink.api.common.functions.{AggregateFunction, ReduceFunction}
import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
import org.apache.flink.api.scala.typeutils.Types
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala.function.WindowFunction
import org.apache.flink.streaming.api.windowing.assigners.{TumblingEventTimeWindows, TumblingProcessingTimeWindows}
import org.apache.flink.streaming.api.windowing.evictors.TimeEvictor
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.windowing.triggers.{ContinuousEventTimeTrigger, Trigger, TriggerResult}
import org.apache.flink.streaming.api.windowing.windows.TimeWindow
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer010
import org.apache.flink.util.Collector
import org.apache