Flink入门(六)Table-Sql

最新推荐文章于 2025-07-20 00:06:45 发布

原创

最新推荐文章于 2025-07-20 00:06:45 发布 · 3.4k 阅读

7 ·

CC 4.0 BY-SA版权

文章标签：

#flink #table-sql

连接kafka

import com.tc.flink.conf.KafkaConfig;
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.Table;
import org.apache.flink.table.api.TableEnvironment;
import org.apache.flink.table.api.java.StreamTableEnvironment;
import org.apache.flink.table.descriptors.Json;
import org.apache.flink.table.descriptors.Kafka;
import org.apache.flink.table.descriptors.Schema;
import org.apache.flink.types.Row;

.....

       	final StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironment();
        StreamTableEnvironment tableEnvironment = TableEnvironment.getTableEnvironment(env);
        Kafka kafkaConnect=new Kafka().version("0.11").topic("topic-test").startFromLatest().property("bootstrap.servers", KafkaConfig.KAFKA_BROKER_LIST).property("group.id", "trafficwisdom-streaming");
        Schema tableSchema=new Schema().field("proctime", Types.SQL_TIMESTAMP).proctime()
                .field("interruptCode", Types.STRING).field("interruptMsg", Types.STRING).field("requestId", Types.STRING).field("transferType", Types.STRING); 
        tableEnvironment.connect(kafkaConnect).withFormat(new Json().failOnMissingField(true).deriveSchema()).withSchema(tableSchema).inAppendMode().registerTableSource("search_log_error");

说明：因为我的kafka消息是json，所以我选取new Json
必须定义Schema，interruptCode、interruptMsg、requestId、transferType且类型全为String类型

kafka消息格式为

{"interruptCode":"B01","interruptMsg":"离线计算-无中转城市","requestId":"565f3d5ef47705fd92be40c70d09c135","transferType":"TF"}
{"interruptCode":"D02","interruptMsg":"回头路规则过滤了所有方案","requestId":"36d1496be24ad63d1d1d6e6bbfaf1e4d","transferType":"TB"}
{"interruptCode":"