[Flink错误总结]flink起不来--＞ Deployment took more than 60 seconds. Please check if the requested resources

最新推荐文章于 2024-02-21 21:59:48 发布
原创 最新推荐文章于 2024-02-21 21:59:48 发布 · 4.3k 阅读 · 2 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#flink #大数据 #big data

本文探讨了当配置高可用性（HA）但未启用Zookeeper时，如何因缺乏资源而引发应用启动失败的问题。解决方法简单明了，只需启动Zookeeper服务，通过`zk.sh start`来修复。

原因:资源不足-->因为配置了HA 但是未开启zookeeper,所以没有资源可以调用,导致启动失败

解决办法-->启动zk即可     zk.sh start 

 

【Flink】Deployment took more than 60 seconds. Please check if the requested resources are available
一杯咖啡半杯糖的博客
12-30 5389
flink任务提交到yarn集群出现如下问题，不断刷屏org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster flink run -yd -m yarn-cluster -c com.test.FlinkCd...
flink on yarn ——报错：Deployment took more than 60 seconds. Please check if the requested resources are
龍承leo
03-15 1万+
报错描述，看着似乎是资源不够使用。 Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster 解决方案：配置yarn-site.xml <property> <name>yarn.scheduler.minimum-allocation-mb</name&gt...
Deployment took more than 60 seconds.check if the requested resources are available in the Yarn clus
m0_58046550的博客
11-29 1150
Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster,yarn内存不足
flink-yarn启动yarn-session时报错Deployment took more than 60 seconds
科小勒的博客
04-14 4848
报错信息 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster 由于flink配置了高可用，所以在启动时需要启动zookeeper，如果不启动zookeeper，高可用则不能使用，会报如上错误。 启动zookeeper再启
yarn 提交任务报错
二十世纪少年的博客
11-07 1883
刚搭的集群，提交flink任务报错INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster。在/etc/hosts中把每台服务器的地址和别名加进去。没有修改hosts文件。
pyflink执行任务问题总结
qq_41895446的博客
12-23 4312
pyflink执行任务问题总结问题：一、Timeout expired while fetching topic metadata二、Deployment took more than 60 seconds三、SyntaxError: invalid syntax四、Failed to deserialize JobGraph五、The cluster does not have the requested resources for the JobManager available!六、Permissio
CDH yarn Fair 队列最大资源使用限制，任务无法提交
永青技术博客
06-30 818
2023-06-29 15:48:20,877 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster 2023-06-29 15:48:21,129 INFO org.apache.flink.yarn.YarnClusterD
flink开发过程中遇到的问题
公众号[大数据技术与应用实战]，分享大数据实战案例
05-30 3963
文章目录类转换异常local class incompatible 类转换异常 cannot be cast to com.google.protobuf.Message Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$RegisterApplicationMasterRequestProto cannot be cast to com.google.protobuf.Messa
[Flink] Flink On Yarn（yarn-session.sh）启动错误
雨潇的专栏
06-05 3497
在Flink上启动 yarn-session.sh时出现 The number of requested virtual cores for application master 1 exceeds the maximum number of virtual cores 0 available in the Yarn Cluster.错误
hadoop 3.x 案例4:Flink启动yarn session问题
只是甲的博客
12-15 1446
Flink 启动yarn session报错 Flink zookeeper配置 Flink 端口号
Flink on yarn 提交任务报错记录
qq_41875667的博客
06-08 828
INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
flink on yarn常见问题及解决方法汇总
fengjian的博客
10-26 5390
持续更新flink on yarn在实际使用中的问题和解决方式，与君共勉
记录一下在个人项目遇到的错误
Anchrew1ow的博客
02-21 308
意思是连不上Kafka，我启动了虚拟机之后，启动项目，运行程序之后报这个错误，一开始没想起来，网上找了一些类似的问题和解决问题，差不多都和IP地址相关，我后来想起来，项目中的application.yml文件中，关于Kafka的bootstrap-servers: 虚拟机地址:9092 配置改为了自己的虚拟机地址，原来是：bootstrap-servers: 127.0.0.1:9092，改过之后，重新启动项目，调试接口，能够正常运行。一般是IP和端口的配置问题，建议查看一下。切换成1.8后不报错！
Flink1.8开发过程遇到的问题
热门推荐
magic_kid_2010的专栏
07-24 1万+
最近在开发flink1.8过程中，提交任务到yarn遇到了一些问题，特将问题及解决办法整理出来。 一、Caused by: java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$RegisterApplicationMasterRequestProto cannot be cast to com....
hadoop yarn 使用_flink on yarn 模式下提示yarn资源不足问题分析
weixin_29208327的博客
01-26 3538
背景在实时计算平台上通过YarnClient向yarn上提交flink任务时一直卡在那里，并在client端一直输出如下日志：(YarnClusterDescriptor.java:1036)- Deployment took more than 60 seconds. Please check if the requested resources are available in th...
生产上的坑才是真的坑 | 盘一盘Flink那些经典线上问题
微信搜：import_bigdata，大数据领域硬核原创作者
03-18 1299
点击上方蓝色字体，选择“设为星标”回复”资源“获取更多资源数据倾斜导致子任务积压业务背景一个流程中，有两个重要子任务：一是数据迁移，将kafka实时数据落Es，二是将kafka数据做窗口聚...
Flink on Yarn no starting application as amIfStarted exceeds amLimit
qq_15138049的博客
11-27 317
yarn no starting application as amIfStarted exceeds amLimit
Flink系列（8）—— Flink 运维问题汇总
shangjg3的博客
08-09 1169
原因 yarn资源不足了。
flink部署-低级错误记录
林子茗的博客
05-09 3697
flink部署时的一些低级错误： 1.提交任务命令打错 2.配了高可用没起zookeeper
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.sport</groupId> <artifactId>bz-sport-realtime</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <!-- 统一版本管理 --> <flink.version>1.20.2</flink.version> <scala.binary.version>2.12</scala.binary.version> <!-- Doris Connector基于Scala 2.12 --> <mysql.cdc.version>2.5.0</mysql.cdc.version> <mysql.driver.version>8.0.33</mysql.driver.version> </properties> <dependencies> <!-- Flink核心依赖 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-base</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-runtime-web</artifactId> <version>${flink.version}</version> </dependency> <!-- Flink Kafka Connector (适配Flink 1.17+) --> <!-- https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>3.4.0-1.20</version> </dependency> <!-- Flink table api 支持 --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java</artifactId> <version>1.20.2</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge</artifactId> <version>1.20.2</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner_${scala.binary.version}</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-common</artifactId> <version>${flink.version}</version> </dependency> <!-- Doris Connector（适配Flink 1.20） --> <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.16</artifactId> <version>25.1.0</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc</artifactId> <version>3.3.0-1.20</version> </dependency> <!--flink cdc 依赖支持--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>3.4.0</version> <!-- <exclusions>--> <!-- <exclusion>--> <!-- <groupId>org.apache.kafka</groupId>--> <!-- <artifactId>kafka-clients</artifactId>--> <!-- </exclusion>--> <!-- </exclusions>--> </dependency> <!--mysql 8 依赖支持--> <dependency> <groupId>com.mysql</groupId> <artifactId>mysql-connector-j</artifactId> <version>8.0.33</version> </dependency> <!-- redis 依赖 --> <dependency> <groupId>org.apache.bahir</groupId> <artifactId>flink-connector-redis_2.12</artifactId> <version>1.1.0</version> </dependency> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>5.2.0</version> </dependency> <!-- rocksdb 状态后端依赖支持 --> <!-- <dependency>--> <!-- <groupId>org.apache.flink</groupId>--> <!-- <artifactId>flink-statebackend-rocksdb</artifactId>--> <!-- <version>1.20.2</version>--> <!-- </dependency>--> <!-- Flink S3插件 --> <!-- <dependency>--> <!-- <groupId>org.apache.flink</groupId>--> <!-- <artifactId>flink-s3-fs-hadoop</artifactId>--> <!-- <version>${flink.version}</version>--> <!-- </dependency>--> <!-- <!– AWS SDK v2 (不依赖Hadoop) –>--> <!-- <dependency>--> <!-- <groupId>software.amazon.awssdk</groupId>--> <!-- <artifactId>s3</artifactId>--> <!-- <version>2.20.56</version>--> <!-- </dependency>--> <!-- 其他工具依赖（按需添加） --> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.14</version> </dependency> <!-- https://mvnrepository.com/artifact/com.alibaba.fastjson2/fastjson2 --> <dependency> <groupId>com.alibaba.fastjson2</groupId> <artifactId>fastjson2</artifactId> <version>2.0.53</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-json</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.30</version> <scope>provided</scope> </dependency> <!-- Hadoop Common --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>3.3.6</version> <!-- 请根据你的 HDFS 集群版本选择，比如 2.7.x, 3.2.x, 3.3.x --> </dependency> <!-- Hadoop HDFS Client --> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs-client</artifactId> <version>3.3.6</version> <!-- 与 hadoop-common 版本保持一致 --> </dependency> <!-- <dependency>--> <!-- <groupId>org.apache.logging.log4j</groupId>--> <!-- <artifactId>log4j-to-slf4j</artifactId>--> <!-- <version>2.14.0</version>--> <!-- </dependency>--> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>3.6.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.2.4</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <!-- 只排除测试依赖或不必要的依赖 --> <exclude>junit:junit</exclude> <!-- <exclude>org.apache.kafka:kafka-clients</exclude>--> </excludes> </artifactSet> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>我的pom是这个
最新发布
08-28
Flink 1.20.2 任务在本地 IDEA 可正常消费 Kafka，但在 YARN 集群上报 `org.apache.kafka.common.KafkaException: Failed to construct kafka consumer` 错误，且提示 `class org.apache.kafka.common.serialization.ByteArrayDeserializer is not an instance of org.apache.kafka.common.serialization.Deserializer`，可从以下几个方面结合 POM 文件进行分析和解决： ### 检查依赖版本兼容性 确保 POM 文件中的 `flink-connector-kafka` 依赖版本与 Flink 版本兼容。对于 Flink 1.20.2，可使用如下依赖： ```xml <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka_2.12</artifactId> <version>1.20.2</version> </dependency> ``` 这里假设使用的 Scala 版本是 2.12。要避免引入不兼容或者冲突的 Kafka 客户端依赖，保证所有 Kafka 相关依赖版本一致。 ### 排除重复依赖 重复的依赖可能会导致类加载冲突，可通过 `mvn dependency:tree` 命令查看依赖树，找出重复的 Kafka 相关依赖并在 POM 文件中排除。例如： ```xml <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka_2.12</artifactId> <version>1.20.2</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> ``` ### 检查类加载顺序 报错可能跟 Flink 的类加载方式有关，可修改 `flink-conf.yml` 中的 `classloader.resolve-order` 参数，将默认的 `child-first` 改成 `parent-first`： ```properties classloader.resolve-order: parent-first ``` ### 检查序列化器配置 确保在代码中正确配置了 Kafka 消费者的反序列化器。示例代码如下： ```java import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.connector.kafka.source.KafkaSource; import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import java.util.Properties; public class FlinkKafkaConsumerExample { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "test-group"); KafkaSource<String> source = KafkaSource.<String>builder() .setBootstrapServers("localhost:9092") .setTopics("test-topic") .setGroupId("test-group") .setStartingOffsets(OffsetsInitializer.earliest()) .setValueOnlyDeserializer(new SimpleStringSchema()) .build(); env.fromSource(source, org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext.class, "Kafka Source") .print(); env.execute("Flink Kafka Consumer Example"); } } ``` ### 检查 YARN 集群环境 确保 YARN 集群上的 Flink 版本和本地开发环境一致，并且检查 YARN 集群上是否已经有 Kafka 客户端的 Jar 包冲突。可以通过 `-Dyarn.provided.lib.dirs` 参数指定 Flink 依赖的 Jar 包路径，避免和集群上已有的 Jar 包冲突。
评论 5
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值