Flink join（流流）详解（一）

关注

原创最新推荐文章于 2026-02-27 19:42:13 发布 2.6k 阅读 AI 写同款· GEO 优化 ›

【Flink知识汇总】【Join专题】 Flink Join 专题Join 的应用场景批Join和流Join的区别Flink 双流Join1. Window Join认识Window JoinFlink APISQL API解决方案的特点解决方案的适用场景2. Interval Join2.1 认识Interval Join2.2 Flink API2.3 SQL API2.4 解决方案的特点2.5 解决方案的适用场景3. Regular Join3.1 认识Regular Join3.2 SQL API3.3 解决方案的特点3.4 解决方案的适阅读详情

本文基于flink 1.11进行测试。

前言

这里所说的join是两个或者多个流的join，涉及流批join的内容或者批批join会另写一篇文章专门说。

Flink的join按照窗口类型分可以分为：Tumbling Window Join、Sliding Window Join和Session Window Join。

按join类型分可以分为join和intervalJoin。前者类似RDBMS中的内连接，interval join使用一个公共键连接两个流的元素（我们现在称它们为A和B），其中流B的元素的时间戳与流A中元素的时间戳之间存在相对时间间隔。

inner join示意图：

interval join示意图：

本文会用到

Flink1.11 事件时间（event time）、watermark、watermarkstrategy使用详细案例文章的知识。
Flink window function及常用窗口算子文章的知识。

代码

创建用于join的两个数据源

package it.kenn.source;

import org.apache.flink.streaming.api.functions.source.SourceFunction;
import scala.Tuple3;

import java.util.Random;

public class ForJoinSource1 implements SourceFunction<Tuple3<String, Long, Double>> {
    boolean flag = true;

    @Override
    public void run(SourceContext<Tuple3<String, Long, Double>> ctx) throws Exception {
        Random random = new Random();
        while (flag) {
            int randInt = random.nextInt(100);
            ctx.collect(new Tuple3<>("S" + randInt, System.currentTimeMillis(), random.nextDouble() * 1000));
            Thread.sleep(30);
        }
    }

    @Override
    public void cancel() {
        flag = false;
    }
}
//-----------------------------------------------------------------------------------
package it.kenn.source;

import org.apache.flink.streaming.api.functions.source.SourceFunction;
import scala.Tuple3;

import java.util.Random;

public class ForJoinSource2 implements SourceFunction<Tuple3<String, Long, Double>> {
    boolean flag = true;

    @Override
    public void run(SourceContext<Tuple3<String, Long, Double>> ctx) throws Exception {
        Random random = new Random();
        while (flag) {
            int randInt = random.nextInt(110);
            ctx.collect(new Tuple3<>("S" + randInt, System.currentTimeMillis(), random.nextDouble() * 1000));
            Thread.sleep(20);
        }
    }

    @Override
    public void cancel() {
        flag = false;
    }
}

测试主程序

package it.kenn.join;

import it.kenn.source.ForJoinSource1;
import it.kenn.source.ForJoinSource2;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.JoinFunction;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.util.Collector;
import scala.Tuple3;

import java.time.Duration;

/**
 * 测试join
 */
//测试主程序
public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamStreamJoinTest joinTest = new StreamStreamJoinTest();
        //设置事件时间
        env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        DataStream<Tuple3<String, Long, Double>> source30 = env.addSource(new ForJoinSource1())
                //指定时间戳和watermark规则，注意要指定两个：forBoundedOutOfOrderness指定watermark生成策略，withTimestampAssigner指定那个字段是事件时间
                .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, Long, Double>>forBoundedOutOfOrderness(Duration.ofMillis(10)).withTimestampAssigner((e, ts) -> e._2()));
        DataStream<Tuple3<String, Long, Double>> source20 = env.addSource(new ForJoinSource2())
                .assignTimestampsAndWatermarks(WatermarkStrategy.<Tuple3<String, Long, Double>>forBoundedOutOfOrderness(Duration.ofMillis(10)).withTimestampAssigner((e, ts) -> e._2()));
        DataStream<Tuple3<String, Long, Double>> tumbJoinedStream = joinTest.tumbJoin(source20, source30);
        DataStream<Tuple3<String, Long, Double>> slidingJoinStream = joinTest.slidingJoin(source20, source30);
        DataStream<Tuple3<String, Long, Double>> intervalJoinStream = joinTest.intervalJoin(source20, source30);
        //对不同join进行测试
        tumbJoinedStream.print();
        env.execute();
    }

inner Tumbling Window Join代码测试

/**
     * inner Tumbling Window Join
     *
     * @param source20
     * @param source30
     * @return
     */
    public DataStream<Tuple3<String, Long, Double>> tumbJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
        DataStream<Tuple3<String, Long, Double>> joinedStream = source20.join(source30)
                .where(e -> e._1().split("-")[1])//左流要join的字段
                .equalTo(e -> e._1().split("-")[1])//右侧流要join的字段
                .window(TumblingEventTimeWindows.of(Time.milliseconds(50)))//指定窗口类型和窗口大小
                //join函数，这里说是join但是跟数据库的join有一些区别，比如下面的逻辑并没有取两个流中的数据，而是比较两个流中数据的大小，只返回某个流中的数据
                .apply(new JoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
                    @Override
                    public Tuple3<String, Long, Double> join(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right) throws Exception {
                        return left._3() > right._3() ? left : right;
                    }
                });
        return joinedStream;
    }

下图是tumble window join的示意图，但是下面join结果有些歧义，像是笛卡尔积。其实只要在join的时候加上where条件就不可能会产生下面笛卡尔积的情况了。

下图还有一个信息点，在最后一个窗口的时候，只有橙色流中有数据，绿色流中并没有数据，那么这个窗口的计算不会被触发。

sliding Join 测试

/**
     * sliding Join 测试
     *
     * @param source20
     * @param source30
     * @return
     */
    public DataStream<Tuple3<String, Long, Double>> slidingJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
        DataStream<Tuple3<String, Long, Double>> joinedStream = source20.join(source30)
                .where(e -> e._1().split("-")[1])//左流要join的字段
                .equalTo(e -> e._1().split("-")[1])//右侧流要join的字段
                .window(SlidingEventTimeWindows.of(Time.milliseconds(50), Time.milliseconds(30)))//指定窗口类型和窗口大小
                .apply(new JoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
                    @Override
                    public Tuple3<String, Long, Double> join(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right) throws Exception {
                        return left._3() > right._3() ? left : right;
                    }
                });
        return joinedStream;
    }

sliding join示意图：

interval join有一个需要注意的特点：有些事件可能在一个滑动窗口中没有被join但是在另外一个滑动窗口中去呗join了。比如上图橙色2号事件，在蓝色窗口中没有与绿色流join，但是在后面的绿色窗口中却与绿色3号join了。

Session Window Join测试

这种方式用到的场景好像不太多，如果哪天我用到了会在这里补上笔记的。

import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.windowing.assigners.EventTimeSessionWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
 
...

DataStream<Integer> orangeStream = ...
DataStream<Integer> greenStream = ...

orangeStream.join(greenStream)
    .where(<KeySelector>)
    .equalTo(<KeySelector>)
    //指定Gap大小
    .window(EventTimeSessionWindows.withGap(Time.milliseconds(1)))
    .apply (new JoinFunction<Integer, Integer, String> (){
        @Override
        public String join(Integer first, Integer second) {
            return first + "," + second;
        }
    });

session window join示意图：

上图是session window示意图。可以看到他的原理是通过两个流的间隔时间划分的窗口，这种窗口的数量非常不稳定。如果流中event间隔一直小于指定的GAP，那么窗口会一直不触发。换句话说，这种窗口的触发相比其他窗口而言比较被动，完全是数据驱动的触发，而不是时间驱动的触发。

interval Join 测试

/**
     * interval Join 测试
     *
     * @param source20
     * @param source30
     * @return
     */
    public DataStream<Tuple3<String, Long, Double>> intervalJoin(DataStream<Tuple3<String, Long, Double>> source20, DataStream<Tuple3<String, Long, Double>> source30) {
        SingleOutputStreamOperator<Tuple3<String, Long, Double>> intervalJoinedStream = source20.keyBy(e -> e._1().split("-")[1])
                .intervalJoin(source30.keyBy(e -> e._1().split("-")[1]))
                .between(Time.milliseconds(-12), Time.milliseconds(9))
                //默认情况下上面的between条件是包含边界的，如果不希望包含边界可以使用下面两个方法去除
                .lowerBoundExclusive()
                .upperBoundExclusive()
                .process(new ProcessJoinFunction<Tuple3<String, Long, Double>, Tuple3<String, Long, Double>, Tuple3<String, Long, Double>>() {
                    @Override
                    public void processElement(Tuple3<String, Long, Double> left, Tuple3<String, Long, Double> right, Context ctx, Collector<Tuple3<String, Long, Double>> out) throws Exception {
                        out.collect(left._3() > right._3() ? left : right);
                    }
                });
        return intervalJoinedStream;
    }

interval join示意图：