实时计算flinksql使用sum(),avg(),max(),min(),over(),WATERMARK示例

最新推荐文章于 2024-08-23 10:05:41 发布

倾一生爱恋换一世纯真

最新推荐文章于 2024-08-23 10:05:41 发布

阅读量1.1k

点赞数 7

文章标签：大数据数据仓库 flink sql

本文链接：https://blog.youkuaiyun.com/weixin_44236758/article/details/135852371

版权

文章详细描述了如何在Flink中创建catalog、database和table，包括从Kafka主题读取数据并将其写入到MySQL表中，以及使用窗口函数进行实时数据分析。展示了如何配置Kafka连接和MySQL连接参数，以及如何定义RowtimeWatermark以处理时间戳事件.

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

-- 创建CATALOG
CREATE CATALOG `FLINK_CATALOG` WITH('type' = 'generic_in_memory');
-- 创建DATABASE
CREATE DATABASE IF NOT EXISTS `FLINK_CATALOG`.`FLINK_DATABASE`;
CREATE TABLE IF NOT EXISTS `FLINK_CATALOG`.`FLINK_DATABASE`.`kafkacs2024` (
  `id` INT,
  `age` INT,
  `int_test` INT,
  `bigint_test` BIGINT,
  `float_test` FLOAT,
  `double_test` DOUBLE,
  `decimal_test` DECIMAL(18, 3),
  `varchar_test` STRING,
  `char_test` STRING,
  `date_test` DATE,
  `datetime_test` VARCHAR,
  `datetimev2_test` TIMESTAMP,
  `namex` STRING,
  `name` STRING,
  `time_test`  AS TO_TIMESTAMP(datetime_test),
  `sex_test` STRING,
  WATERMARK FOR time_test AS time_test - INTERVAL '0' SECOND  --为Rowtime定义Watermark。
) WITH (
  'connector' = 'kafka',
  'topic' = 'kafkacs2024',
  'properties.bootstrap.servers' = '192.168.12.140:9092,192.168.12.139:9092,192.168.12.141:9092',
  'properties.group.id' = 'ea4521bc-4948-4533-ab6e-8aeb0df06be7',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'json'
);
--   引用数据表:  [结果表]
--   type: MYSQL
--   datasourceName: mysql_5
--   tableName: jhhscs
CREATE TABLE IF NOT EXISTS `FLINK_CATALOG`.`FLINK_DATABASE`.`jhhscs1` (
  `id` INT,
    `sex_test` VARCHAR,
    `datetime_test` VARCHAR,
  `count_test` DOUBLE,
  `sum_test` DOUBLE,
  `avg_test` DOUBLE,
  `max_test` DOUBLE,
  `min_test` DOUBLE,
  PRIMARY KEY (`id`) NOT ENFORCED
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:mysql://192.168.12.207:3306/test?useSSL=false&zeroDateTimeBehavior=CO

最低0.47元/天解锁文章