Flink DataStream API 之 windows聚合分类

最新推荐文章于 2024-03-11 16:51:20 发布

Jiny_li

最新推荐文章于 2024-03-11 16:51:20 发布

阅读量1.4k

点赞数 1

CC 4.0 BY-SA版权

文章标签： Flink

本文链接：https://blog.youkuaiyun.com/Jiny_li/article/details/86516507

本文深入探讨了流处理中的两种主要聚合方法：增量聚合与全量聚合。增量聚合在数据流中每接收一条新数据时立即执行计算，适用于实时性要求高的场景；全量聚合则在窗口内所有数据到达后统一进行计算，适合需要对窗口内数据进行复杂操作的情况。文章通过具体示例详细介绍了如何使用Flink API实现这两种聚合方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

基本介绍

增量聚合

•窗口中每进入一条数据，就进行一次计算

•reduce(reduceFunction)

•aggregate(aggregateFunction)

•sum(),min(),max()

全量聚合

•等属于窗口的数据到齐，才开始进行聚合计算【可以实现对窗口内的数据进行排序等需求】

•apply(windowFunction)

•process(processWindowFunction)

processWindowFunction比windowFunction提供了更多的上下文信息。

增量聚合状态变化过程-累加求和（窗口中每进入一条数据，就进行一次计算）

ReduceFunction（增量聚合函数）

DataStream<Tuple2<String, Long>> input = ...;

input
    .keyBy(<key selector>)
    .window(<window assigner>)
    .reduce(new ReduceFunction<Tuple2<String, Long>> {
      public Tuple2<String, Long> reduce(Tuple2<String, Long> v1, Tuple2<String, Long> v2) {
        return new Tuple2<>(v1.f0, v1.f1 + v2.f1);
      }
    });

AggregateFunction（增量聚合函数）

/**
 * The accumulator is used to keep a running sum and a count. The {@code getResult} method
 * computes the average.
 */
private static class AverageAggregate
    implements AggregateFunction<Tuple2<String, Long>, Tuple2<Long, Long>, Double> {
  @Override
  public Tuple2<Long, Long> createAccumulator() {
    return new Tuple2<>(0L, 0L);
  }

  @Override
  public Tuple2<Long, Long> add(Tuple2<String, Long> value, Tuple2<Long, Long> accumulator) {
    return new Tuple2<>(accumulator.f0 + value.f1, accumulator.f1 + 1L);
  }

  @Override
  public Double getResult(Tuple2<Long, Long> accumulator) {
    return ((double) accumulator.f0) / accumulator.f1;
  }

  @Override
  public Tuple2<Long, Long> merge(Tuple2<Long, Long> a, Tuple2<Long, Long> b) {
    return new Tuple2<>(a.f0 + b.f0, a.f1 + b.f1);
  }
}

DataStream<Tuple2<String, Long>> input = ...;

input
    .keyBy(<key selector>)
    .window(<window assigner>)
    .aggregate(new AverageAggregate());

全量聚合状态变化过程-求最大值（等属于窗口的数据到齐，才开始进行聚合计算）

apply

DataStream<Tuple2<String, Long>> input = ...;

input
    .keyBy(<key selector>)
    .window(<window assigner>)
    .apply(new MyWindowFunction());

process

private static class MyProcessWindowFunction
    extends ProcessWindowFunction<SensorReading, Tuple2<Long, SensorReading>, String, TimeWindow> {

  public void process(String key,
                    Context context,
                    Iterable<SensorReading> minReadings,
                    Collector<Tuple2<Long, SensorReading>> out) {
      SensorReading min = minReadings.iterator().next();
      out.collect(new Tuple2<Long, SensorReading>(window.getStart(), min));
  }
}