2、Source算子

Socket源算子

• （1）本地终端输入nc -lk 9527，启动一个Socket源
• （2）然后通过socketTextStream对接Socket源，Demo程序如下：

public class SourceDemo1 {

    public static void main(String[] args) throws Exception {

        // 流式处理环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Socket源算子
        DataStreamSource<String> socketSource = env.socketTextStream("localhost", 9527);

        // 不做任何输出直接打印
        socketSource.print();

        // 执行任务
        env.execute(SourceDemo1.class.getName());

    }

}

Kafka源算子

• 本地启动kafka服务器（先启动zk服务，在启动kafka服务，kafka数据源可以使用kafka自带的控制台生产者）
• 引入Kafka Connector，版本需要与flink的版本对应，最好引入官网文档推荐的依赖

<dependency>
	<groupId>org.apache.flink</groupId>
	<artifactId>flink-connector-kafka_2.11</artifactId>
	<version>1.12.7</version>
</dependency>

• flink程序对接Kafka数据源，Demo程序如下：

public class SourceDemo2 {

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // 设置kafka服务器及topic参数
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "localhost:9092");
        properties.setProperty("group.id", "test");
        DataStream<String> kafkaSource = env
                .addSource(new FlinkKafkaConsumer<>("flinktopic", new SimpleStringSchema(), properties));
        kafkaSource.print();

        env.execute(SourceDemo2.class.getName());
    }

}

自定义算子

• 自定义单并行度算子

自定义单并行度算子通过实现SourceFunction接口实现

• 自定义多并行度算子

自定义多并行度算子通过实现ParallelSourceFunction接口实现

public class SourceDemo3 {

    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // 单并行度Source
        // DataStreamSource<String> singleSource = env.addSource(new MySingleSourceFunction());

        // 多并行度Source
        DataStreamSource<String> parallelSource = env.addSource(new MyParallelSourceFunction()).setParallelism(4);

        System.out.println("Source-Parallelism:" + parallelSource.getParallelism());

        parallelSource.print();

        env.execute(SourceDemo3.class.getName());
    }


    private static class MySingleSourceFunction implements SourceFunction<String> {

        boolean running = true;

        private int index = 1;

        @Override
        public void run(SourceContext<String> ctx) throws Exception {
            while(running) {
                Thread.sleep(1000);
                String data = "消息编号:" + (index++) + ",时间:" + new Date();
                ctx.collect(data);
            }

        }

        @Override
        public void cancel() {
            running = false;
        }
    }

    private static class MyParallelSourceFunction implements ParallelSourceFunction<String> {

        boolean running = true;

        private static AtomicInteger index = new AtomicInteger(1);

        @Override
        public void run(SourceContext<String> ctx) throws Exception {
            while (running) {
                System.out.println("Thread-name:" + Thread.currentThread().getName());
                Thread.sleep(5000);
                int i = index.getAndAdd(1);
                String data = "消息编号:" + i + ",时间:" + new Date();
                ctx.collect(data);
            }

        }

        @Override
        public void cancel() {
            running = false;
        }
    }

}

参考资料

• kafka v1.12文档