7. Flink on yarn模式部署以及flink和hive的集成

1. Flink Yarn 模式高可用性配置(自己通过源码编译的Flink)

本次采用自编译（基于Flink-1.10，hadoop-2.9.2）搭建集群

• flink-shaded-hadoop-2-uber官方链接

• 参考链接

  • Flink高可用集群搭建
  • Flink JobManager HA高可用
  • flink on yarn模式下两种提交job方式

• 启动集群时遇到的错误

  • 错误一

    java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
    	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:197)
    	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.<init>(FlinkYarnSessionCli.java:173)
    	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:836)
    

  • 解决方案
    在 flink-shaded-9.0/flink-shaded-hadoop-2-uber/pom.xml 中的 dependencyManagement 标签中添加如下依赖Apache Commons CLI » 1.4；重新编译mvn clean install -DskipTests -Drat.skip=true -Pvendor-repos -Dhadoop.version=2.9.2;最后再重新编译flink源码mvn clean package -T 4 -Dfast -Drat.skip=true -Pinclude-hadoop -Dhadoop.version=2.9.2 -Dmaven.compile.fork=true -DskipTests -Dscala-2.11
    

  • 错误二

    java.io.IOException: Could not create FileSystem for highly available storage path (hdfs://xiaofan/flink/ha/default)
            at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:103)
            at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:89)
            at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:125)
            at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:132)
            at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
            at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
            at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
            at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
            at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
    Caused