flink1.10三节点集群standalone模式搭建

最新推荐文章于 2025-09-30 04:14:50 发布

原创

最新推荐文章于 2025-09-30 04:14:50 发布 · 3.4k 阅读

1 ·

CC 4.0 BY-SA版权

本文详细介绍了如何搭建Flink的standalone模式集群，包括配置JDK环境、SSH免密登录、配置文件调整等步骤。特别关注了动态添加TaskManager节点的过程，以及在不修改配置文件的情况下进行集群扩展的方法。

各台机器上提前准备jdk1.8以及上的java环境，并且配置ssh免密登录。

集群环境

flink1:172.21.89.128	jobmanager
flink2:172.21.89.129	taskmanager
flink3:172.21.89.130	taskmanager

在flink1上做flink配置，主要是flink-conf.yaml、masters和slaves

flink-conf.yaml：

jobmanager.rpc.address: flink1
# 每个taskmanager机器提供的slot数量
taskmanager.numberOfTaskSlots: 2
# 默认并行度
parallelism.default: 2
# 临时文件存储路径。需要提前创建，否则启动集群会报错。
io.tmp.dirs: /root/flink/tmp

slaves：

flink2

配置完之后通过scp -r ./flink-1.10.0 flink2:/opt将flink文件传输到flink2。之后就可以在flink1上启动集群：bin/start-cluster.sh

我这里因为只在flink1上提前创建了临时文件目录/root/flink/tmp，而flink2上没有，所以启动时，jobmanager启动成功，但taskmanager启动失败：

2020-05-19 16:03:32,276 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2020-05-19 16:03:32,645 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner       - TaskManager initialization failed.
java.lang.Exception: unable to establish the security context
        at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:73)
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:319)
        at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
Caused by: java.lang.RuntimeException: unable to generate a JAAS configuration file
        at org.apache.flink.runtime.security.modules.JaasModule.generateDefaultConfigFile(JaasModule.java:170)
        at org.apache.flink.runtime.security.modules.JaasModule.install(JaasModule.java:94)
        at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:67)
        ... 2 more
Caused by: java.nio.file.NoSuchFileException: /root/flink/tmp/jaas-7614411253117836328.conf
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
        at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
        at java.nio.file.Files.createTempFile(Files.java:852)
        at org.apache.flink.runtime.security.modules.JaasModule.generateDefaultConfi