flink1.17入门

已于 2023-11-23 22:12:14 修改
阅读量494 收藏 2
点赞数
分类专栏： 大数据 文章标签： flink
于 2023-11-01 19:55:44 首次发布
原文链接：https://mp.youkuaiyun.com/mp_blog/creation/editor/134168911
版权

前置条件，先把hadoop学会

hadoop3入门(1)-优快云博客

创建普通的maven项目

pom

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>1.17.0</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients</artifactId>
    <version>1.17.0</version>
</dependency>

 流式处理

就是一条数据一条数据的处理

 分词统计案例

有界流-读文件

创建aa.txt,按照空格区分

aa bb cc
dd ee ff
aa bb dd
ee cc dd
package com.example.demoscala.controller;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * 读文件 有界流(程序直接结束)
 * 来一条读取一条
 */
public class Test {
    public static void main(String[] args) throws Exception {
        //创建执行环境
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //读取数据: 从文件读
        DataStreamSource<String> dataStreamSource = exec.readTextFile("wenjian/aa.txt");
        //处理数据: 切分 转换 分组 聚合
        //切分转换
        SingleOutputStreamOperator<Tuple2<String, Integer>> aa = dataStreamSource.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            //匿名内部类 实现切分转换 返回元组
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                //按照空格 切分
                String[]sp=s.split(" ");
                for (String x : sp) {
                    //转换成2元组(word,1) 第一个是单词，第二个是1
                    Tuple2<String, Integer> tp = Tuple2.of(x, 1);
                    //通过采集器 向下游发送数据
                    collector.collect(tp);
                }
            }
        });

        //分组
        KeyedStream<Tuple2<String, Integer>, String> bb = aa.keyBy(new KeySelector<Tuple2<String, Integer>, String>() {

            @Override
            public String getKey(Tuple2<String, Integer> v) throws Exception {
                //按照单词进行分组
                return v.f0;
            }
        });
        //聚合         对单词进行求和
        SingleOutputStreamOperator<Tuple2<String, Integer>> cc = bb.sum(1);
        //输出数据
        cc.print();
        //执行
        exec.execute();
    }
}

无界流-读socket

package com.example.demoscala.controller;

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

/**
 * 读socket 无界流(长连接 不停止程序)
 *
 */
public class Test {
    public static void main(String[] args) throws Exception {
       //创建执行环境
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //读取数据 socket
        DataStreamSource<String> dataStreamSource = exec.socketTextStream("192.168.10.102", 9999);
        //处理数据 切分 转换 分组 聚合
        SingleOutputStreamOperator<Tuple2<String, Integer>> aa = dataStreamSource.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                //按照空格切分
                String[] sp = s.split(" ");
                for (String x : sp) {
                    //转换成二元组
                    Tuple2<String, Integer> y = Tuple2.of(x, 1);
                    collector.collect(y);
                }
            }
        });
        //分组
        KeyedStream<Tuple2<String, Integer>, String> bb = aa.keyBy(new KeySelector<Tuple2<String, Integer>, String>() {
            @Override
            public String getKey(Tuple2<String, Integer> stringIntegerTuple2) throws Exception {
                return stringIntegerTuple2.f0;
            }
        });
        //聚合
        SingleOutputStreamOperator<Tuple2<String, Integer>> cc = bb.sum(1);
        //输出
        cc.print();
        //执行
        exec.execute();
    }
}

在102的服务器上,输入nc -lk 9999端口号

在idea的控制台就能看到处理的数据，并且程序不会关闭

集群角色


客户端：Java代码，提交给JobManager
JobManager: 对作业进行调度，获取要执行的作业，进一步转换，分发任务给TaskManager
TaskManager: 数据的处理操作都是他来做 

ip/域名角色
192.168.10.102（hadoop102）

JobManager

TaskManager

192.168.10.103（hadoop103）

TaskManager
192.168.10.104（hadoop104）TaskManager

安装部署 

下载包

https://dlcdn.apache.org/flink/flink-1.17.1/flink-1.17.1-bin-scala_2.12.tgz

然后放入3台机器中的/opt/module目录下，并解决

tar -zxvf flink-1.17.1-bin-scala_2.12.tgz

修改集群配置

cd /opt/module/flink-1.17.1/conf

vi flink-conf.yaml

在102机器的配置

# JobManager节点地址.
jobmanager.rpc.address: hadoop102
jobmanager.bind-host: 0.0.0.0
rest.address: hadoop102
rest.bind-address: 0.0.0.0
# TaskManager节点地址.需要配置为当前机器名
taskmanager.bind-host: 0.0.0.0
taskmanager.host: hadoop102 

在103机器的配置

# JobManager节点地址.
jobmanager.rpc.address: hadoop102
jobmanager.bind-host: 0.0.0.0
rest.address: hadoop102
rest.bind-address: 0.0.0.0
# TaskManager节点地址.需要配置为当前机器名
taskmanager.bind-host: 0.0.0.0
taskmanager.host: hadoop103 

在104机器的配置

# JobManager节点地址.
jobmanager.rpc.address: hadoop102
jobmanager.bind-host: 0.0.0.0
rest.address: hadoop102
rest.bind-address: 0.0.0.0
# TaskManager节点地址.需要配置为当前机器名
taskmanager.bind-host: 0.0.0.0
taskmanager.host: hadoop104 

配置workers，3台机器都配置

vi workers

hadoop102
hadoop103
hadoop104 

在3台机器上配置主节点端口号

vi masters

 hadoop102:8081

在102机器上启动flink集群

cd /opt/module/flink-1.17.1/bin

./start-cluster.sh

访问ui界面

http://hadoop102:8081/

 

向集群提交作业 

添加pom打包插件

   <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                    <exclude>org.slf4j:*</exclude>
                                    <exclude>log4j:*</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <!-- Do not copy the signatures in the META-INF folder.
                                    Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers combine.children="append">
                                <transformer
                                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer">
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

上传jar包到这里 

点击详情，输入Test的路径，点击提交

 

 在102的机器上发送消息，在ui界面的stdout,就能看到计算的结果，点刷新按钮

部署模式


会话模式：先启动一个集群，保持一个会话，在这个会话中通过客户端提交作业
单作业模式：为了更好的隔离资源，可以为每个提交的作业启动一个集群，这就是单作业模式
应用模式： 不需要客户端，直接把应用提交到jobmanager上运行，为每一个提交的应用单独启动一个jobmanager,也就是创建一个集群，执行结束后，jobmanager也就关闭了 

历史服务器

当8081的端口挂掉之后，可以在8082的端口看到历史的任务信息

先在hdfs中创建logs目录

然后再3台机器上面修改配置

cd /opt/module/flink-1.17.1/conf

vi flink-conf.yaml

jobmanager.archive.fs.dir: hdfs://hadoop102:8020/logs
historyserver.web.address: hadoop102
historyserver.web.port: 8082
historyserver.archive.fs.dir: hdfs://hadoop102:8020/logs
historyserver.archive.fs.refresh-interval: 5000
 

然后进入flink的bin目录下启动历史服务器

./historyserver.sh start 

关闭历史服务器可以使用

./historyserver.sh stop

当8081挂掉之后，就会在hdfs中存储历史的信息

访问8082就可以看到历史的信息

http://hadoop102:8082 

DataStream Api 

DataStream Api是Flink的核心层Api,一个flink程序，就是对dataStream 的各种转换，由这几步构成:执行环境，读取数据源，转换操作，输出，执行

源算子

从集合中读取数据

public class Test {
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        List<Integer> list= Arrays.asList(1,2,1,3,2);
        //从集合中读取数据
        DataStreamSource<Integer>ds=exec.fromCollection(list);
        //输出
        ds.print();
        //执行
        exec.execute();
    }
}

从文件中读取数据

之前text已经过时了，需要先引入pom

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-files</artifactId>
    <version>1.17.0</version>
</dependency>

public class Test {
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();

        //从文件中读取数据
        FileSource<String> fileSource = FileSource.forRecordStreamFormat(new TextLineInputFormat()
                        , new Path("wenjian/aa.txt"))
                .build();
        //输出
        exec.fromSource(fileSource, WatermarkStrategy.noWatermarks(),"资源名称")
                .print();


        //执行
        exec.execute();
    }
}

从kafka读取

引入pom,并提前创建好topic

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka</artifactId>
    <version>1.17.0</version>
</dependency>

 kafka一定要提前安装好

package com.example.demo7;

import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;


public class Test {
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();

        KafkaSource<String> kafkaSource = KafkaSource.<String>builder()
                //kafka的ip和端口号
                .setBootstrapServers("192.168.10.102:9092")
                //主题的名称 提前创建好
                .setTopics("zhuti")
                //组id
                .setGroupId("zuid")
                .setStartingOffsets(OffsetsInitializer.latest())
                //反序列化 字符串
                .setValueOnlyDeserializer(new SimpleStringSchema())
                //建造对象
                .build();
        //读取资源 并输出
        exec.fromSource(kafkaSource, WatermarkStrategy.noWatermarks(),"资源名称")
                .print();

        //执行
        exec.execute();
    }
}

kafka生产者发送消息 

idea控制台接收消息

从数据生成器读取数据 

引入pom

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-datagen</artifactId>
    <version>1.17.0</version>
</dependency>

package com.example.demo7;


import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.connector.source.util.ratelimit.RateLimiterStrategy;
import org.apache.flink.connector.datagen.source.DataGeneratorSource;
import org.apache.flink.connector.datagen.source.GeneratorFunction;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class Test {
    public static void main(String[] args) throws Exception {

        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //设置并行1 单线程
        exec.setParallelism(1);
        DataGeneratorSource<String> dataGeneratorSource=new DataGeneratorSource<>(new GeneratorFunction<Long, String>() {
            @Override
            public String map(Long aLong) throws Exception {
                return "内容:"+aLong;
            }
        },
                //最多到10,可以设置无限大
                10,
                //限流策略 每秒有多少个许可 可以进入
                RateLimiterStrategy.perSecond(1),
                Types.STRING);
        //读取资源数据 并打印
        exec.fromSource(dataGeneratorSource, WatermarkStrategy.noWatermarks(),"资源名称").print();

        //执行
        exec.execute();
    }
}

 

基本转换算子

map


主要用于将数据进行转换，生成新的数据 ,只能返回一条数据

package com.example.demo7;


import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(new User("1","张三"),
                new User("2","李四"));
        //只返回id
        SingleOutputStreamOperator<String> mp = s.map(x -> x.getId());
        mp.print();

        exec.execute();
    }
}

 

flatMap

可以返回多条数据，也可以把一行数据拆分

package com.example.demo7;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(new User("1","张三"),
                new User("2","李四"));

        SingleOutputStreamOperator<String> mp = s.flatMap(new FlatMapFunction<User, String>() {
            @Override
            public void flatMap(User user, Collector<String> collector) throws Exception {
                collector.collect(user.getId());
                collector.collect("可以输出多条信息");
            }
        });
        mp.print();

        exec.execute();
    }
}

filter

数据过滤，只要等于true的 都打印出来

package com.example.demo7;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(new User("1","张三"),
                new User("2","李四"));

        //id=2的打印出来
        SingleOutputStreamOperator<User> filter = s.filter(x -> "2".equals(x.getId()));
        filter.print();
        exec.execute();
    }
}

package com.example.demo7;

public class User {

    private String id;

    private String name;

    public User(String id, String name) {
        this.id = id;
        this.name = name;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    @Override
    public String toString() {
        return "User{" +
                "id='" + id + '\'' +
                ", name='" + name + '\'' +
                '}';
    }
}

聚合算子

聚合算子必须和keyby一起使用

sum

package com.example.demo7;

public class User {

    private String id;

    private Integer age;

    public User(){}

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public Integer getAge() {
        return age;
    }

    public void setAge(Integer age) {
        this.age = age;
    }

    public User(String id, Integer age) {
        this.id = id;
        this.age = age;
    }

    @Override
    public String toString() {
        return "User{" +
                "id='" + id + '\'' +
                ", age=" + age +
                '}';
    }
}

package com.example.demo7;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(
                new User("1",10),
                new User("2",2),
                new User("1",20),
                new User("2",3));
        //按照id 分组 相当于mysql的group by
        KeyedStream<User, String> userStringKeyedStream = s.keyBy(new KeySelector<User, String>() {
            @Override
            public String getKey(User user) throws Exception {
                return user.getId();
            }
        });
        //对age进行累加求和   
        SingleOutputStreamOperator<User> aa = userStringKeyedStream.sum("age");
        //打印
        aa.print();
        
        exec.execute();
    }
}

reduce

package com.example.demo7;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.util.Collector;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(
                new User("1",10),
                new User("2",2),
                new User("1",20),
                new User("2",3));
        //按照id 分组 相当于mysql的group by
        KeyedStream<User, String> userStringKeyedStream = s.keyBy(new KeySelector<User, String>() {
            @Override
            public String getKey(User user) throws Exception {
                return user.getId();
            }
        });
        //对age进行累加求和
        SingleOutputStreamOperator<User> reduce = userStringKeyedStream.reduce(new ReduceFunction<User>() {
            @Override
            public User reduce(User user, User t1) throws Exception {
                //累加 求和 组成新的对象
                return new User(user.getId(), user.getAge() + t1.getAge());
            }
        });
        //打印
        reduce.print();

        exec.execute();
    }
}

 

随机分区

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        //设置并行度 2个线程
        exec.setParallelism(2);
        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(
                new User("1",10),
                new User("2",2),
                new User("1",20),
                new User("2",3));
        //随机分区
        s.shuffle().print();

        exec.execute();
    }
}

 

分流 

就是把一条河流中的数据，开一个口子，让其中一部分的数据，分开走

package com.example.demo7;


import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.KeyedStream;
import org.apache.flink.streaming.api.datastream.SideOutputDataStream;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;
import org.apache.flink.util.OutputTag;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();

        //从对象中读取数据
        DataStreamSource<User> s = exec.fromElements(
                new User("1",10),
                new User("2",2),
                new User("1",20),
                new User("3",3));

        SingleOutputStreamOperator<String> mp = s.map(new MapFunction<User, String>() {
            @Override
            public String map(User user) throws Exception {
                return user.getId();
            }
        });
        OutputTag<String>s1=new OutputTag<>("1", Types.STRING);
        OutputTag<String>s2=new OutputTag<>("2", Types.STRING);
        SingleOutputStreamOperator<String> p = mp.process(new ProcessFunction<String, String>() {
            @Override
            public void processElement(String s, ProcessFunction<String, String>.Context context, Collector<String> collector) throws Exception {
                    if(s.equals("1")){
                        //放入1流
                        context.output(s1,s+":aaaa");
                    }else if(s.equals("2")){
                        //放入2流
                        context.output(s2,s+":cccc");
                    }else {
                        //放入主流
                        collector.collect(s);
                    }
            }
        });
        //获取测流
        SideOutputDataStream<String> a = p.getSideOutput(s1);
        SideOutputDataStream<String> c = p.getSideOutput(s2);

        //打印主流
        p.print();
        //打印测流
        a.print();
        c.print();

        exec.execute();
    }
}

合流 

union

合并2个流的数据成一条流

package com.example.demo7;


import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<Integer> aa = exec.fromElements(1, 2, 3);
        DataStreamSource<Integer> bb = exec.fromElements(4, 5, 6);
        //合并流
        aa.union(bb).print();


        exec.execute();
    }
}

connect

合并2个不同类型的流

package com.example.demo7;


import org.apache.flink.streaming.api.datastream.ConnectedStreams;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.co.CoProcessFunction;
import org.apache.flink.util.Collector;

public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<Integer> aa = exec.fromElements(1, 2, 3);
        DataStreamSource<String> bb = exec.fromElements("a", "b", "c");
        //合并流
        ConnectedStreams<Integer, String> connect = aa.connect(bb);
        //把2个流转换成string输出
        SingleOutputStreamOperator<String> process = connect.process(new CoProcessFunction<Integer, String, String>() {
            @Override
            public void processElement1(Integer integer, CoProcessFunction<Integer, String, String>.Context context, Collector<String> collector) throws Exception {
                collector.collect(integer.toString());
            }

            @Override
            public void processElement2(String s, CoProcessFunction<Integer, String, String>.Context context, Collector<String> collector) throws Exception {
                collector.collect(s);
            }
        });
        process.print();


        exec.execute();
    }
}

 

输出算子 

输出到文件

package com.example.demo7;


import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.serialization.SimpleStringEncoder;
import org.apache.flink.connector.file.sink.FileSink;
import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.resourcemanager.WorkerResourceSpecFactory;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.WindowedStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig;
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner;

import java.time.ZoneId;


public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<String> aa = exec.fromElements("aa", "vb", "cc");
        //必须开启
        exec.enableCheckpointing(2000, CheckpointingMode.EXACTLY_ONCE);
        //输出到文件系统
        FileSink<String> f = FileSink.<String>forRowFormat(new Path("f:/aa"), new SimpleStringEncoder<>("UTF-8"))
                //设置生成文件的后缀
                .withOutputFileConfig(OutputFileConfig.builder().withPartSuffix(".txt").build())
                //按照目录分桶 每个小时一个目录
                .withBucketAssigner(new DateTimeBucketAssigner<>("yyyy-MM-dd HH", ZoneId.systemDefault()))
                .build();
        aa.sinkTo(f);
        exec.execute();
    }
}

输出到kafka 

package com.example.demo7;


import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.serialization.SimpleStringEncoder;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.connector.base.DeliveryGuarantee;
import org.apache.flink.connector.file.sink.FileSink;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.connector.kafka.sink.KafkaSink;
import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.resourcemanager.WorkerResourceSpecFactory;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.datastream.WindowedStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig;
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner;
import org.apache.kafka.clients.producer.ProducerConfig;

import java.time.ZoneId;


public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();

        //必须开启
        exec.enableCheckpointing(2000, CheckpointingMode.EXACTLY_ONCE);

        SingleOutputStreamOperator<String> aa = exec.socketTextStream("192.168.10.102",9999);

        //指定kafka的ip和端口
        KafkaSink<String> kafkaSink = KafkaSink.<String>builder().setBootstrapServers("192.168.10.102:9092")
                //指定序列化器 指定topic名称 具体的序列化
                .setRecordSerializer(KafkaRecordSerializationSchema.<String>builder()
                        .setTopic("zhuti")
                        .setValueSerializationSchema(new SimpleStringSchema())
                        .build())
                //写到kafka的一致性级别：精准一次 至少一次
                .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
                //如果是精准一次 必须设置 事务的前缀
                .setTransactionalIdPrefix("aaaaaa-")
                //如果是精准一次 必须设置事务的超时时间 大于Checkpoint间隔 小于max 15分钟
                .setProperty(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 10 * 60 * 1000 + "")
                .build();
        aa.sinkTo(kafkaSink);
        exec.execute();
    }
}

通过socket发送数据，kafka消费者接收数据 

 

输出到mysql

创建表

CREATE TABLE `user1` (
  `id` int NOT NULL,
  `name` varchar(2) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci DEFAULT NULL,
  `age` int NOT NULL,
  `asd` tinyint DEFAULT NULL,
  KEY `asd` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

package com.example.demo7;


import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.serialization.SimpleStringEncoder;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.connector.base.DeliveryGuarantee;
import org.apache.flink.connector.file.sink.FileSink;
import org.apache.flink.connector.jdbc.JdbcConnectionOptions;
import org.apache.flink.connector.jdbc.JdbcExecutionOptions;
import org.apache.flink.connector.jdbc.JdbcSink;
import org.apache.flink.connector.jdbc.JdbcStatementBuilder;
import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema;
import org.apache.flink.connector.kafka.sink.KafkaSink;
import org.apache.flink.core.fs.Path;
import org.apache.flink.runtime.resourcemanager.WorkerResourceSpecFactory;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.datastream.WindowedStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig;
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.DateTimeBucketAssigner;
import org.apache.kafka.clients.producer.ProducerConfig;

import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.time.ZoneId;


public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();

        //必须开启
        exec.enableCheckpointing(2000, CheckpointingMode.EXACTLY_ONCE);

        SingleOutputStreamOperator<User> aa = exec.socketTextStream("192.168.10.102",9999)
                .map(new MapFunction<String, User>() {
                    @Override
                    public User map(String s) throws Exception {
                        String []sp=s.split(",");
                        return new User(sp[0],Integer.parseInt(sp[1]));
                    }
        });


        //INSERT INTO `dmg1`.`user1` (`id`, `name`, `age`, `asd`) VALUES (1, '张三', 100, 127);
        SinkFunction<User> jdbcSink = JdbcSink.sink(
                "insert into user1 values(?,?,?,?)",
                new JdbcStatementBuilder<User>() {
                    @Override
                    public void accept(PreparedStatement preparedStatement, User user) throws SQLException {
                        //每收到一条WaterSensor，如何去填充占位符
                        preparedStatement.setString(1, user.getId());
                        preparedStatement.setString(2, "张三");
                        preparedStatement.setInt(3, user.getAge());
                        preparedStatement.setInt(4, 1);
                    }
                },
                JdbcExecutionOptions.builder()
                        // 重试次数
                        .withMaxRetries(3)
                        // 批次的大小：条数
                        .withBatchSize(100)
                        // 批次的时间
                        .withBatchIntervalMs(3000)
                        .build(),
                new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
                        .withUrl("jdbc:mysql://localhost:3306/dmg1?serverTimezone=Asia/Shanghai&useUnicode=true&characterEncoding=UTF-8")
                        .withUsername("root")
                        .withPassword("123456")
                        .withConnectionCheckTimeoutSeconds(60) // 重试的超时时间
                        .build()
        );
        aa.addSink(jdbcSink);
        exec.execute();
    }
}

 引入pom

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.27</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-jdbc</artifactId>
    <version>1.17-SNAPSHOT</version>
</dependency>

<repositories>
    <repository>
        <id>apache-snapshots</id>
        <name>apache snapshots</name>
        <url>https://repository.apache.org/content/repositories/snapshots/</url>
    </repository>
</repositories>

maven仓库设置 添加!apache-snapshots

从socket输入数据，添加到mysql中 

窗口

 增量聚合函数

package com.example.demo7;


import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;


public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        SingleOutputStreamOperator<User> aa = exec.socketTextStream("192.168.10.102",9999)
                .map(new MapFunction<String, User>() {
                    @Override
                    public User map(String s) throws Exception {
                        String []sp=s.split(",");
                        return new User(sp[0],Integer.parseInt(sp[1]));
                    }
        });
        //安装id 分组
        aa.keyBy(x->x.getId())
                //设置滚动事件 时间窗口  5秒
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                //增量聚合
                .reduce(new ReduceFunction<User>() {
                    @Override
                    public User reduce(User user, User t1) throws Exception {
                        //2个age相加 组成新的对象
                        return new User(user.getId(),user.getAge()+t1.getAge());
                    }
                })
                .print();
        exec.execute();
    }
}

5秒打印一次数据，累加之前的数据 

全窗口函数

全窗口函数需要先收集窗口中的数据，并在内部缓存起来，等到窗口要输出结果的时候，在取出来进行计算

package com.example.demo7;


import org.apache.commons.lang3.time.DateFormatUtils;
import org.apache.flink.api.common.functions.AggregateFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;


public class Test {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment exec = StreamExecutionEnvironment.getExecutionEnvironment();
        SingleOutputStreamOperator<User> aa = exec.socketTextStream("192.168.10.102",9999)
                .map(new MapFunction<String, User>() {
                    @Override
                    public User map(String s) throws Exception {
                        String []sp=s.split(",");
                        return new User(sp[0],Integer.parseInt(sp[1]));
                    }
        });
        //安装id 分组
        aa.keyBy(x->x.getId())
                //设置滚动事件 时间窗口  5秒
                .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
                //全窗口函数
                        .process(new ProcessWindowFunction<User, Object, String, TimeWindow>() {
                            @Override
                            public void process(String s, ProcessWindowFunction<User, Object, String, TimeWindow>.Context context, Iterable<User> iterable, Collector<Object> collector) throws Exception {
                                long count=iterable.spliterator().estimateSize();
                                long start=context.window().getStart();
                                long end=context.window().getEnd();
                                String kaishi= DateFormatUtils.format(start,"yyyy-MM-dd HH:mm:ss");
                                String jieshu= DateFormatUtils.format(end,"yyyy-MM-dd HH:mm:ss");
                                collector.collect("当前key:"+s+",开始时间:"+kaishi+",结束时间:"+jieshu+",总数:"+count);
                            }
                        }).print();
        exec.execute();
    }
}

只统计这5秒内的数据，之前的数据 不会累计

Flink Sql

在102机器的flink的bin目录下 启动客户端

./sql-client.sh

创建数据库

create database dmg;

 进入数据库

use dmg;

创建表

CREATE TABLE sink (
    id INT, 
    ts BIGINT, 
    vc INT
) WITH (
'connector' = 'print'
);

 创建数据源生成器

CREATE TABLE source ( 
    id INT, 
    ts BIGINT, 
    vc INT
) WITH ( 
    'connector' = 'datagen', 
    'rows-per-second'='1', 
    'fields.id.kind'='random', 
    'fields.id.min'='1', 
    'fields.id.max'='10', 
    'fields.ts.kind'='sequence', 
    'fields.ts.start'='1', 
    'fields.ts.end'='1000000', 
    'fields.vc.kind'='random', 
    'fields.vc.min'='1', 
    'fields.vc.max'='100'
);

设置显示模式

SET sql-client.execution.result-mode=tableau;

查看数据源生成器 

【Flink-1.17-教程】-【一】Flink概述、Flink快速入门
weixin_53543905的博客
01-15 1612
对于Flink而言，流才是整个处理逻辑的底层核心，所以流批统一之后的 DataStream API 更加强大，可以直接处理批处理和流处理的所有场景。 下面我们就针对不同类型的输入数据源，用具体的代码来实现流处理。 Flink 还具有一个类型提取系统，可以分析函数的输入和返回类型，自动获取类型信息，从而获得对应的序列化器和反序列化器。但是，由于 Java 中`泛型擦除`的存在，在某些特殊情况下（比如 Lambda 表达式中），自动提取的信息是不够精细的——只告诉 Flink 当前的元素由“船头、船身、船尾”构
Flink入门(四) -- Flink中的窗口
最新发布
zzy66666c的博客
08-13 1313
窗口通常指的是建筑物中用来采光和通风的构件，即窗户。这是“窗口”一词最直接和常见的含义。
Flink1.17入门
x3303208457的博客
10-18 127
（5）高可用: 本身高可用的设置，加上与K8s，YARN 和 Mesos 的紧密集成，再加上从故障中快速恢复和动态扩展任务的能力，Flink能做到以及少的停机时间7*24全天运行。（2）结果的准确性：Flink提供了事件时间和处理时间语义。对于乱序事件流事件时间，事件时间 语义依然能提供一致且准确的结果。（4）可以连接到最常用的外部系统，如Kafka、Hive、JDBC、HDFS、Redis等。我们处理数据的目标是：低延迟、高吞吐、结果的准确性和良好的容错性。
Flink 1.17 Flink-SQL-Gateway HiveServer2 源码分析
EdwardWang_的博客
03-30 1286
Flink 1.17 Flink-SQL-Gateway HiveServer2
Flink1.17总结
weixin_53932655的博客
11-09 1002
用户自定义函数（user-defined function，UDF），即用户可以根据自身需求，重新实现算子的逻辑。用户自定义函数分为：函数类、匿名函数、富函数类。
深入解读 Flink 1.17
Ververica的博客
05-24 1345
摘要：本文整理自阿里云技术专家，Apache Flink PMC Member & Committer、Flink CDC Maintainer 徐榜江(雪尽) 在深入解读 Flink 1.17 Meetup 的分享。
flink入门文档.pdf
08-16
Flink 入门文档 Flink 是一个大数据处理框架，具有批流一体、高容错、高吞吐、低延迟、大规模计算、多平台部署等核心特点。本文档将对 Flink 的核心概念、架构、API 层次进行详细介绍。 1. 核心概念 Flink 是一个...
flink零基础入门.pdf
06-09
Apache Flink 进阶（一）：Runtime 核心机制剖析 4 Apache Flink 进阶（二）：时间属性深度解析 18 Apache Flink 进阶（三）：Checkpoint 原理剖析与应用实践 30 Apache Flink 进阶（四）：Flink on Yarn/K8s 原理...
flink-1.7-中文文档(官网英文翻译过来).pdf
05-13
flink中文文档 非常详细 快速上手入门 5分钟上手开发 流计算入门权威资料 上手flink 必看！
深入解读Flink1.17
阳龙生
04-27 632
flink1.17有什么新特性，来看看吧
Apache Flink 1.17
qq_40822132的博客
06-08 1408
Flink 1.17 版本完成了 7 个 FLIP，累计贡献者 170+，解决 600+Issue 以及 1100+Commits，整体来看是一个较大的版本。
Flink1.17学习笔记（自用）
m0_64735594的博客
07-01 1272
作业提交给Yarn的ResourceManager，选择一个节点NodeManager启动一个容器，容器里运行ApplicationMaster（JobManager），JobManager里面启动分发器和资源管理器，分发器启动JobMaster生成逻辑流图StreamGraph，经过算子链的优化生成作业流图JobGraph，将作业流图并行化展开生成执行流图ExecutionGraph。end=start+窗口长度。任务槽的共享组，在同一个作业中，不同任务节点的并行子任务，就可以放到同一个slot上执行。
Flink1.17实战教程（第一篇：概念、部署、架构）
moments的博客
11-29 2759
Flink官网Flimk核心目标，是“数据流上的有状态计算具体说明：Apache Flink是一个框架和分布式处理引擎，用于对无界和有界数据流进行有状态计算。（1）无界流数据有定义流的开始，但没有定义流的结束；它们会无休止的产生数据；无界流的数据必须持续处理，即数据被摄取后需要立刻处理我们不能等到所有数据都到达再处理，因为输入是无限的。（2）有界流数据有定义流的开始，也有定义流的结束；有界流可以在摄取所有数据后再进行计算；有界流所有数据可以被排序，所以并不需要有序摄取；
flink java 并行度_Apache Flink 1.3-SNAPSHOT 中文文档: 并行执行
weixin_39623350的博客
03-02 215
Application DevelopmentManaging Execution并行执行并行执行本节描述了如何在Flink中配置程序的并行执行。一个Flink程序由多个任务组成(变换/操作符, 数据源和 sinks)。一个任务被切分为多个并行的实例来执行，而每一个并行的实例处理任务输入数据的一个子集。一个任务的并行实例数目就被称为该任务的并行度。如果你想使用savepoints，你应该同时考虑设...
【Flink-1.17-教程】-【二】Flink 集群搭建、Flink 部署、Flink 运行模式
weixin_53543905的博客
01-16 4167
YARN 上部署的过程是：客户端把 Flink 应用提交给 Yarn 的 ResourceManager，Yarn 的 ResourceManager 会 向 Yarn 的 NodeManager 申 请 容 器 。 在 这 些 容 器 上 ， Flink 会 部 署 JobManager 和 TaskManager 的实例，从而启动集群。Flink 会根据运行在 JobManger 上的作业所需要的 Slot 数量动态分配 TaskManager 资源。
flink
宋小生-中间件
04-25 409
海量数据实时分析引擎Apache Flink 简介 当系统出现大量或者重大的错误却不被
尚硅谷大数据Flink1.17实战教程-笔记01【Flink概述、Flink快速上手】
upward
06-14 4987
尚硅谷大数据Flink1.17实战教程-笔记01【Flink概述、Flink快速上手】
Flink源码学习二 用户代码逻辑计划生成 1.17分支
weixin_47019045的博客
10-11 1227
接上节flink调用用户代码的main方法后，用户代码中一般会有如下获取flink环境的代码这里获取的环境再命令行客户的执行这个方法是就已经初始化好了，这个env里面封装了一些用户环境配置、streaming执行配置等。
在JDK17尝鲜Flink1.17
云来山更佳，云去山如画
07-03 5564
在JDK17尝鲜Flink1.17 前言 还没玩明白老版本，Flink1.17就来了！！！总还是要向前看的。。。 根据官网文档：https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/try-flink/local_installation/ Flink runs on all UNIX-like environments, i.e. Linux, Mac OS X, and Cygwin (for Windows). You nee
flink1.17 sql
07-28
Flink 1.17引入了对SQL一些新的功能和改进。这些改进主要包括以下几个方面： 1. 支持更多的SQL语法：Flink 1.17增加了对SQL标准的支持，包括更多的SQL语法和内置函数。这使得用户可以更方便地编写复杂的SQL查询。 2. 改进的查询优化器：Flink 1.17引入了改进的查询优化器，可以更高效地执行SQL查询。这个优化器可以自动推断查询中的关联关系，并选择合适的执行计划来提高查询性能。 3. 动态表连接：Flink 1.17引入了动态表连接功能，可以在运行时动态地连接两个表。这使得用户可以在查询中根据不同的条件动态地进行表连接操作，从而更灵活地处理数据。 4. 支持逻辑删除：Flink 1.17支持逻辑删除功能，可以在删除数据时将其标记为已删除而不是物理删除。这使得用户可以在查询中过滤掉已删除的数据，从而更好地管理数据的生命周期。 5. 改进的事件时间处理：Flink 1.17改进了事件时间处理的性能和可靠性。它引入了更高效的事件时间窗口处理机制，并提供了更多的事件时间特性，如事件时间戳提取和水印生成等。 这些是Flink 1.17中关于SQL的一些新功能和改进。希望能对你有所帮助！如果你有更多的问题，请继续提问。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值