部署seatunnel2.3.8分离模式集群

部署seatunnel web参考：SeaTunnel Web1.0.0安装_plugindiscoveryutil.getallconnectors-优快云博客

配置：两台centos服务器，2master2worker

一、下载包

v2.3.8

[bin] apache-seatunnel-2.3.8-bin.tar.gz

将包上传到master节点和worker节点所在服务器，路径：/data/seatunnel/apache-seatunnel-2.3.8-bin.tar.gz

# 解压
tar -zxvf apache-seatunnel-2.3.8-bin.tar.gz
mv apache-seatunnel-2.3.8  seatunnel-2.3.8 

二、设置环境变量

vim /etc/profile

# 新增以下环境变量配置
export SEATUNNEL_HOME=/data/seatunnel/seatunnel-2.3.8
export PATH=$PATH:$SEATUNNEL_HOME/bin


# 生效
source /etc/profile

三、配置 JVM

1、Master节点配置：$SEATUNNEL_HOME/config/jvm_master_options

# JVM Heap
-Xms2g
-Xmx3g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=3g

# G1GC
-XX:+UseG1GC

2、Worker节点配置：$SEATUNNEL_HOME/config/jvm_worker_options

# JVM Heap
-Xms2g
-Xmx3g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=3g

# G1GC
-XX:+UseG1GC

四、配置 Checkpoint和Imap(Master节点）

1、配置seatunnel.yaml 

以下配置使用oss作为checkpoint等数据存储，使用的阿里云的oss，需要开通oss_hdfs功能

# master节点的以下配置
vim /data/seatunnel/seatunnel-2.3.8/config/seatunnel.yaml 

seatunnel:
  engine:
    history-job-expire-minutes: 1440
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          storage.type: oss
          oss.bucket: oss://test_bucket/ # 替换成自己的
          fs.oss.accessKeyId: accessKeyId # 替换成自己的
          fs.oss.accessKeySecret: accessKeySecret # 替换成自己的
          fs.oss.endpoint: oss_hdfs_endpoint # 替换成自己的
          fs.oss.credentials.provider: org.apache.hadoop.fs.aliyun.oss.AliyunCredentialsProvider

map:
  engine*:
    map-store:
      enabled: true
      initial-mode: EAGER
      factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
      properties:
        type: hdfs
        namespace: /seatunnel/imap # 替换成自己的
        clusterName: seatunnel-cluster
        storage.type: oss
        block.size: 256
        oss.bucket: oss://test_bucket/ # 替换成自己的
        fs.oss.accessKeyId: accessKeySecret # 替换成自己的
        fs.oss.accessKeySecret: accessKeySecret # 替换成自己的
        fs.oss.endpoint: oss_hdfs_endpoint # 替换成自己的

2、依赖jar包上传到阿里云oss

根目录下没有lib目录，新建一个

上传以下几个包到lib目录

aliyun-sdk-oss-3.13.2.jar
hadoop-aliyun-3.3.6.jar
jdom2-2.0.6.jar
netty-buffer-4.1.89.Final.jar 
netty-common-4.1.89.Final.jar
seatunnel-hadoop3-3.1.4-uber.jar

五、配置TCP

1、修改master节点

vim /data/seatunnel/seatunnel-2.3.8/config/hazelcast-master.yaml

hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - 192.168.1.1:5801 # 要修改的，master节点
          - 192.168.1.1:5802 # 要修改的，worker节点
          - 192.168.1.2:5801 # 要修改的，master节点
          - 192.168.1.2:5802 # 要修改的，worker节点
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 2
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100

2、修改worker节点

vim /data/seatunnel/seatunnel-2.3.8/config/hazelcast-worker.yaml

hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - 192.168.1.1:5801 # 要修改的，master节点
          - 192.168.1.1:5802 # 要修改的，worker节点
          - 192.168.1.2:5801 # 要修改的，master节点
          - 192.168.1.2:5802 # 要修改的，worker节点
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50
    hazelcast.heartbeat.failuredetector.type: phi-accrual
    hazelcast.heartbeat.interval.seconds: 2
    hazelcast.max.no.heartbeat.seconds: 180
    hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
    hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
    hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100

六、启动

1、启动Master节点

./bin/seatunnel-cluster.sh -d -r master

2、启动Worker节点

./bin/seatunnel-cluster.sh -d -r worker

七、测试

# 执行以下命令测试是否可以启动 
bin/seatunnel.sh --config config/v2.batch.config.template

八、不同的数据源依赖的包(Master节点）

1、上传mysql驱动

下载mysql-connector-java-8.0.28.jar上传到：${SEATUNNEL_HOME}/lib/目录下

2、上传连接器

 下载connector-开头的连接器包：https://repo.maven.apache.org/maven2/org/apache/seatunnel/

上传到：${SEATUNNEL_HOME}/connectors/目录下