flink对watermark的处理

峰二爷

于 2021-12-31 14:43:04 发布

阅读量1.1k

点赞数

分类专栏：笔记文章标签：大数据 big data flink

本文链接：https://blog.youkuaiyun.com/bwf317/article/details/122256029

版权

笔记专栏收录该内容

53 篇文章

订阅专栏

DataStream<WaterPowerData> waterPowerStream = inputStream.map(new MapFunction<String, WaterPowerData>() {
            @Override
            public WaterPowerData map(String s) throws Exception {
                WaterPowerData waterPowerData = JSONObject.parseObject(s, WaterPowerData.class);
                return waterPowerData;
            }
        }).assignTimestampsAndWatermarks(new AssignerWithPeriodicWatermarks<WaterPowerData>() {
            private long currentTimestamp = Long.MIN_VALUE;
            private final long maxTimeLag = 5000;

            @Override
            public long extractTimestamp(WaterPowerData waterPowerData, long l) {
                long timestamp = waterPowerData.getOutputTime();
                currentTimestamp = Math.max(timestamp, currentTimestamp);
                return timestamp;
            }

            @Nullable
            @Override
            public Watermark getCurrentWatermark() {
                return new Watermark(currentTimestamp == Long.MIN_VALUE ? Long.MIN_VALUE : currentTimestamp - maxTimeLag);
            }
        });