flink集群各种部署方式

flink部署方式

无资源管理standalone

原生无平台

1. Session Mode

   # (1) Start Cluster
   $ ./bin/start-cluster.sh
   # (2) You can now access the Flink Web Interface on http://localhost:8081
   # (3) Submit example job
   $ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
   # (4) Stop the cluster again
   $ ./bin/stop-cluster.sh
   

2. Application Mode

   $ cp ./examples/streaming/TopSpeedWindowing.jar lib/
   Then, we can launch the JobManager:
   $ ./bin/standalone-job.sh start --job-classname org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
   The web interface is now available at localhost:8081. However, the application won’t be able to start, because there are no TaskManagers running yet:
   $ ./bin/taskmanager.sh start
   Note: You can start ~~~~~~~~multiple TaskManagers, if your application needs more resources.
   Stopping the services is also supported via the scripts. Call them multiple times if you want to stop multiple instances, or use stop-all:
   $ ./bin/taskmanager.sh stop
   $ ./bin/standalone-job.sh stop
   

3. Per-Job Mode
   不支持

运行在docker

1. Session Mode

   #-------------------------命令行---------------------------
   #设置变量,创建docker-network
   $ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
   $ docker network create flink-network
   #创建jobmanager
   $ docker run \
     --rm \
     --name=jobmanager \
     --network flink-network \
     --publish 8081:8081 \
     --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
     flink:1.12.2-scala_2.11 jobmanager
   #创建taskmanager
   $ docker run \
     --rm \
     --name=taskmanager \
     --network flink-network \
     --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
     flink:1.12.2-scala_2.11 taskmanager
   #进入jobmanager,run
   $ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
   #-------------------------docker-compose---------------------------
   version: "2.2"
       services:
         jobmanager:
           image: flink:1.12.2-scala_2.11
           ports:
             - "8081:8081"
           command: jobmanager
           environment:
             - |
               FLINK_PROPERTIES=
               jobmanager.rpc.address: jobmanager
       
         taskmanager:
           image: flink:1.12.2-scala_2.11
           depends_on:
             - jobmanager
           command: taskmanager
           scale: 1
           environment:
             - |
               FLINK_PROPERTIES=
               jobmanager.rpc.address: jobmanager
               taskmanager.numberOfTaskSlots: 2
   

2. Application Mode
   通过运行flink docker的standalone-job就会以application模式启动Jobmanager container
   还得把程序包挂载到容器内部的/opt/flink/usrlib

   -----------------------------直接命令挂载方式--------------------------
   $ FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
   $ docker network create flink-network
   
   $ docker run \
       --mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
       --mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
       --rm \
       --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
       --name=jobmanager \
       --network flink-network \
       flink:1.12.2-scala_2.11 standalone-job \
       --job-classname com.job.ClassName \
       [--job-id <job id>] \
       [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
       [job arguments]
   
   $ docker run \
       --mount type=bind,src=/host/path/to/job/artifacts1,target=/opt/flink/usrlib/artifacts1 \
       --mount type=bind,src=/host/path/to/job/artifacts2,target=/opt/flink/usrlib/artifacts2 \
       --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
       flink:1.12.2-scala_2.11 taskmanager
   
   -----------------------------Dockerfile生成镜像方式--------------------------
   FROM flink
   ADD /host/path/to/job/artifacts/1 /opt/flink/usrlib/artifacts/1
   ADD /host/path/to/job/artifacts/2 /opt/flink/usrlib/artifacts/2
   
   $ docker build --tag flink_with_job_artifacts .
   $ docker run \
       flink_with_job_artifacts standalone-job \
       --job-classname com.job.ClassName \
       [--job-id <job id>] \
       [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] \
       [job arguments]
   
   $ docker run flink_with_job_artifacts taskmanager
   

   也可以使用docker-compose部署application

    version: "2.2"
    services:
      jobmanager:
        image: flink:1.12.2-scala_2.11
        ports:
          - "8081:8081"
        command: standalone-job --job-classname com.job.ClassName [--job-id <job id>] [--fromSavepoint /path/to/savepoint [--allowNonRestoredState]] [job arguments]
        volumes:
          - /host/path/to/job/artifacts:/opt/flink/usrlib
        environment:
          - |
            FLINK_PROPERTIES=
            jobmanager.rpc.address: jobmanager
            parallelism.default: 2
    
      taskmanager:
        image: flink:1.12.2-scala_2.11
        depends_on:
          - jobmanager
        command: taskmanager
        scale: 1
        volumes:
          - /host/path/to/job/artifacts:/opt/flink/usrlib
        environment:
          - |
            FLINK_PROPERTIES=
            jobmanager.rpc.address: jobmanager
            taskmanager.numberOfTaskSlots: 2
            parallelism.default: 2
   

   可以定制化创建一个镜像
   https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/docker.html#advanced-customization
3. Per-Job Mode
   不支持

运行在k8s

ConfigMap,Service通用yaml
https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/resource-providers/standalone/docker.html#advanced-customization

#flink-configuration-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: flink-config
  labels:
    app: flink