Flink 双流Join

JNWsong

已于 2024-12-06 10:28:44 修改

阅读量1.4k

点赞数

分类专栏： flink 文章标签： flink 大数据

于 2023-04-26 11:29:14 首次发布

本文链接：https://blog.youkuaiyun.com/java_creatMylief/article/details/130358848

版权

flink 专栏收录该内容

16 篇文章

订阅专栏

Reguler Join (常规join)

inner join

left join / right join

INNER/LEFT/RIGHT/FULL OUTER

DataStream API（函数编程）

window Join

join

对处于同一窗口的数据进行join

时间类型：processTime、eventTime

问题：1、不在同一窗口的数据无法join，

2、只能inner join

import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;

...

DataStream<Integer> orangeStream = ...
DataStream<Integer> greenStream = ...

orangeStream.join(greenStream)
    .where(<KeySelector>) // 左侧key值
    .equalTo(<KeySelector>) // 右侧key值
    .window(SlidingEventTimeWindows.of(Time.milliseconds(2) /* size */, Time.milliseconds(1) /* slide */)) // 开窗方式 tumbing/sliding/session
    .apply (new JoinFunction<Integer, Integer, String> (){
        @Override
        public String join(Integer first, Integer second) {
            return first + "," + second;
        }
    });

coGroup

coGroup是join的底层方法，通过coGroup可以实现inner/left/right/full 四种join

时间类型：processTime、eventTime

问题：不在同一窗口的数据无法join

latedata.coGroup(stream)
    .where(a->a.getStr("a"))
    .equalTo(a->a.getStr("a"))
    .window(TumblingEventTimeWindows.of(Time.seconds(10)))
    .apply(new CoGroupFunction<JSONObject, JSONObject, Object>() {
           @Override
           public void coGroup(Iterable<JSONObject> iterable, Iterable<JSONObject> iterable1, Collector<Object> collector) throws Exception {
                            
     }
})

interval Join

为了解决window join的问题：处于不同窗口的数据无法join

时间类型：eventTime

interval join ：根据左流的数据的时间点，左右各等待一段右流时间，在此范围内进行join

问题：只能是以左流为时间线，因此只支持inner join

import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.functions.co.ProcessJoinFunction;
import org.apache.flink.streaming.api.windowing.time.Time;

...

DataStream<Integer> orangeStream = ...
DataStream<Integer> greenStream = ...

orangeStream
    .keyBy(<KeySelector>)
    .intervalJoin(greenStream.keyBy(<KeySelector>))
    .between(Time.milliseconds(-2), Time.milliseconds(1))
    .process (new ProcessJoinFunction<Integer, Integer, String(){

        @Override
        public void processElement(Integer left, Integer right, Context ctx, Collector<String> out) {
            out.collect(first + "," + second);
        }
    });

Table API（flink sql)

Reguler Join (常规join)

默认没有时间范围，全局都可以join，时间长了状态一定会爆炸

必须设置数据过期时间

tableEnv.getConfig().setIdleStateRetention(xx)

设置过期时间后以后，四种join 数据过期方式各有不同

inner join

inner join 左流右流，创建后进入过期倒计时

SELECT *
FROM Orders
INNER JOIN Product
ON Orders.product_id = Product.id

left join / right join

left: 左流创建后进入过期倒计时，但是成功join一次后，就会重置过期时间

right: 右流创建后进入过期倒计时，但是成功join一次后，就会重置过期时间

SELECT *
FROM Orders
LEFT JOIN Product
ON Orders.product_id = Product.id

SELECT *
FROM Orders
RIGHT JOIN Product
ON Orders.product_id = Product.id

full join

左、右流创建后进入过期倒计时，但是成功join一次后，就会重置过期时间

SELECT *
FROM Orders
FULL OUTER JOIN Product
ON Orders.product_id = Product.id

interval join

作为DataStreamApi升级版的interval join，sql版本的支持处理时间语义和事件事件语义

SELECT *
FROM Orders o, Shipments s
WHERE o.id = s.order_id
AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time

lookup join

效果等同于cdc,但是每次过来一条数据都会去数据库进行一次查询关联、效率差

但是可以设置缓存机制，如果用过一次后会缓存指定的时间，但是在缓存期间内就不会实时同步mysql的数据了。此时就和regular join 一样了

因此lookup join 试用场景为字典数据需要变化，但是变化的时间不需要实时变化，有点延迟也可以。

Lookup join 一定要用左关联（left join）,lookup join 是上述 interval join的变种，如果使用 (inner ) join 且关联维度恰好不存在，会导致主流丢失，一定要注意

关键语句

FOR SYSTEM_TIME AS OF o.proc_time

lookup.cache.max-rows

optional

(none)

Integer

The max number of rows of lookup cache, over this value, the oldest rows will be expired. Lookup cache is disabled by default. See the following Lookup Cache section for more details

最多缓存多少条

lookup.cache.ttl

optional

(none)

Duration

The max time to live for each rows in lookup cache, over this time, the oldest rows will be expired. Lookup cache is disabled by default. See the following Lookup Cache section for more details.

缓存数据ttl

1 DAY

1 HOUR

CREATE TEMPORARY TABLE Orders (
  id INT,
  order_id INT,
  total INT,
  proc_time as procetime()
) WITH (
  'connector' = 'kafka',
  ...
);

-- Customers is backed by the JDBC connector and can be used for lookup joins
CREATE TEMPORARY TABLE Customers (
  id INT,
  name STRING,
  country STRING,
  zip STRING
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:mysql://mysqlhost:3306/customerdb',
  'table-name' = 'customers',
  'lookup.cache.max-rows' = '10',
  'lookup.cache.ttl' = '1 hour'
);

-- enrich each order with customer information
SELECT o.order_id, o.total, c.country, c.zip
FROM Orders AS o
-- 最好用LEFT JOIN，INNER JOIN的场景很少
LEFT JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
    ON o.customer_id = c.id;

-- flink1.16+ 新提供的hints，lookup 重试策略，如果关联不上，重试600次，每次间隔1秒，这种场景下，会阻塞主流，重试600次后，才能接受下一跳
SELECT /*+ LOOKUP('table'='c', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='1s', 'max-attempts'='600') */
o.order_id, o.total, c.country, c.zip
FROM orders AS o
JOIN customers
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;

-- 异步重试，但是得'output-mode'='allow_unordered'，允许乱序
SELECT /*+ LOOKUP('table'='c', 'retry-predicate'='lookup_miss', 'output-mode'='allow_unordered', 'retry-strategy'='fixed_delay', 'fixed-delay'='1s', 'max-attempts'='600') */
o.order_id, o.total, c.country, c.zip
FROM orders AS o
JOIN customers /*+ OPTIONS('lookup.async'='true', 'lookup.async-thread-number'='16') */
FOR SYSTEM_TIME AS OF o.proc_time AS c
ON o.customer_id = c.id;

Look up join |Apache Flink --- Hints

Window Join

窗口join，必须对表进行TVF开窗才能使用

table(tumple(table tablegreen,descriptor(rt),interval '5' minutes))

时间类型：processTime、eventTime

INNER/LEFT/RIGHT/FULL OUTER

SELECT ...
FROM L [LEFT|RIGHT|FULL OUTER] JOIN R -- L and R are relations applied windowing TVF
ON L.window_start = R.window_start AND L.window_end = R.window_end AND ...