Apache Flink连载（十）：Flink集群基础环境搭建-JDK及MySQL搭建

原创 已于 2023-12-28 00:25:38 修改 · 968 阅读 · 24 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#apache #flink #java

于 2023-12-09 04:00:00 首次发布
本文详细介绍了如何在Apache Flink集群环境中搭建JDK和MySQL。首先，按照步骤在所有节点上安装JDK8，并配置环境变量。接着，在选定的节点上安装MySQL5.7，包括在CentOS7中处理安装问题和启动服务。然后，对MySQL进行配置，包括设置root密码、修改密码策略和编码格式。最后，提供了忘记MySQL密码的解决办法。

🏡 个人主页：IT贫道_大数据OLAP体系技术栈,Apache Doris,Clickhouse 技术-优快云博客

 🚩 私聊博主：加入大数据技术讨论群聊，获取更多大数据资料。

 🔔 博主个人B栈地址：豹哥教你大数据的个人空间-豹哥教你大数据个人主页-哔哩哔哩视频

目录

1. 安装JDK

2. 安装MySQL

2.1 节点划分

2.2 安装MySQL

2.3 ​​​​​​​配置MySQL

​​​​​​​2.4 Mysql密码忘记处理

​​​​​​​1. 安装JDK

按照以下步骤在各个节点上安装JDK8。

1) 各个节点创建/software目录，上传并安装jdk8 rpm包

rpm -ivh /software/jdk-8u181-linux-x64.rpm

以上命令执行完成后，会在每台节点的/user/java下安装jdk8。

2) 配置jdk环境变量

在每台节点上配置jdk的环境变量：
了解本专栏 订阅专栏 解锁全文 超级会员免费看
记录 Flink jdbc、mysql-cdc 连接 mysql8 碰到的适配问题
主要分享大数据相关的知识，如Spark、Hudi
04-29 1397
记录 Flink jdbc、mysql-cdc 连接 mysql8 碰到的小问题
Zookeeper+Hadoop+Spark+Flink+Kafka+Hbase+Hive 完全分布式高可用集群搭建(保姆级超详细含图文) 附安装包网盘下载
Mr.L-OAM的博客
07-05 2万+
Hadoop+Spark+Flink+Zookeeper+Kafka+Hbase+Hive完全分布式高可用集群搭建 Hadoop高可用集群搭建 Spark高可用集群搭建 Flink高可用集群搭建 Zookeeper高可用集群搭建 Kafka高可用集群搭建 Hbase高可用集群搭建 Hive高可用集群搭建 mysql搭建 jdk部署 大数据集群环境搭建
Flink1.11.0+Kafka2.3.0+MySQL8的快速安装部署
xiaopeigen的专栏
07-10 1296
环境准备 本实战演示环节需要安装到Linux虚拟机的服务，包括： Flink 集群：用来运行 Flink 任务。 Kafka 集群：用来作为数据源。 MySQL 数据库：用来作为结果表。 虚拟机最好是CenOs7以上的版本，这样会支持docker，后续一些组件可以采用docker的方式部署 特别说明 如果我们平时是用本机window编写IDEA，可以不用着急安装虚拟机上面的Flink Flink 集群安装 下载 Flink 1.11.0安装包并解压（解压目录flink-1.11.0）：h..
Flink+Kafka+Mysql(8.0)
u014293455的博客
11-06 1537
Flink版本：1.14.6Mysql版本：8.0JDK版本：1.8本实例主要实现功能如下：模拟消息生成->Kafka->Flink->Mysql其中Flink做数据流收集并定时批量写入到Mysql本例使用Intellij IDEA作为项目开发的IDE。
Flink（62）：Flink中通用MySQLUtil工具类
yang_shibiao的博客
05-18 3224
此时一个较为方便的工具类就能很方便的使用，能达到节省开发时间、减小开发难度等目的。upsertByPrimaryKey(String tableName, boolean underScoreToCamel, Object object, String... fields) ： 根据主键使用ON DUPLICATE KEY UPDATE语法对对应的表中的数据进行更新，也可以对传入的数据中的字段是否下划线驼峰命名转换，并将传入的字段集合作为update条件（字段集合必须在object中有数据）
Flink读取Kafka 消息并批量写入到 MySQL8.0
xiaopeigen的专栏
07-09 2444
前言说明 环境搭建可参考《kafka+flink集成实例》 本实例主要实现功能如下： 模拟消息生成->Kafka->Flink->Mysql 其中Flink做数据流收集并定时批量写入到Mysql 环境依赖 本例使用Intellij IDEA作为项目开发的IDE。 首先创建Maven project，group为’com.zuoan’，artifact id为‘flink-kafka-mysql’，version为‘1.0-SNAPSHOT’。整个项目结构如图所示： 特..
oscar@oscardeMacBook-Pro bz-sport-realtime % mvn dependency:tree [INFO] Scanning for projects... [INFO] [INFO] --------------------< com.sport:bz-sport-realtime >--------------------- [INFO] Building bz-sport-realtime 1.0-SNAPSHOT [INFO] from pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [WARNING] 2 problems were encountered while building the effective model for org.apache.yetus:audience-annotations:jar:0.5.0 during dependency collection step for project (use -X to see details) [INFO] [INFO] --- dependency:3.7.0:tree (default-cli) @ bz-sport-realtime --- [INFO] com.sport:bz-sport-realtime:jar:1.0-SNAPSHOT [INFO] +- org.apache.flink:flink-java:jar:1.20.2:compile [INFO] | +- org.apache.commons:commons-lang3:jar:3.12.0:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile [INFO] | +- com.twitter:chill-java:jar:0.7.6:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.7.36:compile [INFO] | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] +- org.apache.flink:flink-core:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-core-api:jar:1.20.2:compile [INFO] | | \- org.apache.flink:flink-metrics-core:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-annotations:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-shaded-asm-9:jar:9.5-17.0:compile [INFO] | +- org.apache.flink:flink-shaded-jackson:jar:2.14.2-17.0:compile [INFO] | +- org.snakeyaml:snakeyaml-engine:jar:2.6:compile [INFO] | +- org.apache.commons:commons-text:jar:1.10.0:compile [INFO] | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile [INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile [INFO] | | \- org.objenesis:objenesis:jar:2.1:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.26.0:compile [INFO] | \- org.apache.flink:flink-shaded-guava:jar:31.1-jre-17.0:compile [INFO] +- org.apache.flink:flink-connector-base:jar:1.20.2:compile [INFO] +- org.apache.flink:flink-streaming-java:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-file-sink-common:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-runtime:jar:1.20.2:compile [INFO] | | +- org.apache.flink:flink-rpc-core:jar:1.20.2:compile [INFO] | | +- org.apache.flink:flink-rpc-akka-loader:jar:1.20.2:compile [INFO] | | +- org.apache.flink:flink-queryable-state-client-java:jar:1.20.2:compile [INFO] | | +- org.apache.flink:flink-hadoop-fs:jar:1.20.2:compile [INFO] | | +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.7.1-17.0:compile [INFO] | | +- org.javassist:javassist:jar:3.24.0-GA:compile [INFO] | | \- tools.profiler:async-profiler:jar:2.9:compile [INFO] | \- org.apache.flink:flink-connector-datagen:jar:1.20.2:compile [INFO] +- org.apache.flink:flink-clients:jar:1.20.2:compile [INFO] | +- org.apache.flink:flink-optimizer:jar:1.20.2:compile [INFO] | +- commons-cli:commons-cli:jar:1.5.0:compile [INFO] | \- org.apache.flink:flink-datastream:jar:1.20.2:compile [INFO] | \- org.apache.flink:flink-datastream-api:jar:1.20.2:compile [INFO] +- org.apache.flink:flink-runtime-web:jar:1.20.2:compile [INFO] | \- org.apache.flink:flink-shaded-netty:jar:4.1.91.Final-17.0:compile [INFO] +- org.apache.flink:flink-connector-kafka:jar:3.4.0-1.20:compile [INFO] | +- com.google.guava:guava:jar:32.1.2-jre:compile [INFO] | | +- com.google.guava:failureaccess:jar:1.0.1:compile [INFO] | | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile [INFO] | | +- org.checkerframework:checker-qual:jar:3.33.0:compile [INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.18.0:compile [INFO] | | \- com.google.j2objc:j2objc-annotations:jar:2.8:compile [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.15.2:compile [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.2:compile [INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.2:compile [INFO] | \- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.2:compile [INFO] +- org.apache.flink:flink-table-api-java:jar:1.20.2:compile [INFO] +- org.apache.flink:flink-table-api-java-bridge:jar:1.20.2:provided [INFO] | \- org.apache.flink:flink-table-api-bridge-base:jar:1.20.2:provided [INFO] +- org.apache.flink:flink-table-planner_2.12:jar:1.20.2:compile [INFO] | +- org.immutables:value:jar:2.8.8:compile [INFO] | +- org.immutables:value-annotations:jar:2.8.8:compile [INFO] | +- org.codehaus.janino:commons-compiler:jar:3.1.10:compile [INFO] | +- org.codehaus.janino:janino:jar:3.1.10:compile [INFO] | +- org.apache.flink:flink-scala_2.12:jar:1.20.2:compile [INFO] | | +- org.scala-lang:scala-reflect:jar:2.12.7:compile [INFO] | | +- org.scala-lang:scala-library:jar:2.12.7:compile [INFO] | | +- org.scala-lang:scala-compiler:jar:2.12.7:compile [INFO] | | | \- org.scala-lang.modules:scala-xml_2.12:jar:1.0.6:compile [INFO] | | \- com.twitter:chill_2.12:jar:0.7.6:compile [INFO] | \- org.apache.flink:flink-table-runtime:jar:1.20.2:compile [INFO] | \- org.apache.flink:flink-cep:jar:1.20.2:compile [INFO] +- org.apache.flink:flink-table-common:jar:1.20.2:compile [INFO] | \- com.ibm.icu:icu4j:jar:67.1:compile [INFO] +- org.apache.doris:flink-doris-connector-1.16:jar:25.1.0:compile [INFO] +- org.apache.flink:flink-connector-jdbc:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-core:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-cratedb:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-db2:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-mysql:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-oceanbase:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-oracle:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-postgres:jar:3.3.0-1.20:compile [INFO] | +- org.apache.flink:flink-connector-jdbc-sqlserver:jar:3.3.0-1.20:compile [INFO] | \- org.apache.flink:flink-connector-jdbc-trino:jar:3.3.0-1.20:compile [INFO] +- org.apache.flink:flink-connector-mysql-cdc:jar:3.4.0:compile [INFO] | +- org.apache.flink:flink-connector-debezium:jar:3.4.0:compile [INFO] | | +- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] | | \- io.debezium:debezium-embedded:jar:1.9.8.Final:compile [INFO] | | +- org.apache.kafka:connect-api:jar:3.2.0:compile [INFO] | | | \- javax.ws.rs:javax.ws.rs-api:jar:2.1.1:runtime [INFO] | | +- org.apache.kafka:connect-runtime:jar:3.2.0:compile [INFO] | | | +- org.apache.kafka:connect-transforms:jar:3.2.0:compile [INFO] | | | +- org.apache.kafka:kafka-tools:jar:3.2.0:runtime [INFO] | | | | \- net.sourceforge.argparse4j:argparse4j:jar:0.7.0:runtime [INFO] | | | +- org.bitbucket.b_c:jose4j:jar:0.7.9:runtime [INFO] | | | +- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.12.6:runtime [INFO] | | | | +- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.12.6:runtime [INFO] | | | | \- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.6:runtime [INFO] | | | | \- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:runtime [INFO] | | | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.34:runtime [INFO] | | | | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:runtime [INFO] | | | | | \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:runtime [INFO] | | | | \- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:runtime [INFO] | | | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:runtime [INFO] | | | | \- org.glassfish.hk2:hk2-locator:jar:2.6.1:runtime [INFO] | | | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:runtime [INFO] | | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:runtime [INFO] | | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:runtime [INFO] | | | +- javax.activation:activation:jar:1.1.1:compile [INFO] | | | +- org.eclipse.jetty:jetty-servlets:jar:9.4.44.v20210927:runtime [INFO] | | | | \- org.eclipse.jetty:jetty-continuation:jar:9.4.44.v20210927:runtime [INFO] | | | +- org.eclipse.jetty:jetty-client:jar:9.4.44.v20210927:runtime [INFO] | | | +- org.reflections:reflections:jar:0.9.12:runtime [INFO] | | | \- org.apache.maven:maven-artifact:jar:3.8.4:runtime [INFO] | | | \- org.codehaus.plexus:plexus-utils:jar:3.3.0:runtime [INFO] | | +- org.apache.kafka:connect-json:jar:3.2.0:compile [INFO] | | \- org.apache.kafka:connect-file:jar:3.2.0:compile [INFO] | +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | | +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | | +- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] | | | \- org.antlr:antlr4-runtime:jar:4.8:compile [INFO] | | +- com.zendesk:mysql-binlog-connector-java:jar:0.27.2:compile [INFO] | | \- mysql:mysql-connector-java:jar:8.0.28:compile [INFO] | +- org.apache.flink:flink-cdc-common:jar:3.4.0:compile [INFO] | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile [INFO] | +- com.zaxxer:HikariCP:jar:4.0.3:compile [INFO] | \- org.apache.flink:flink-cdc-runtime:jar:3.4.0:compile [INFO] +- com.mysql:mysql-connector-j:jar:8.0.33:compile [INFO] | \- com.google.protobuf:protobuf-java:jar:3.21.9:compile [INFO] +- org.apache.bahir:flink-connector-redis_2.12:jar:1.1.0:compile [INFO] | +- org.apache.flink:flink-streaming-java_2.12:jar:1.14.5:compile [INFO] | | \- org.apache.flink:flink-shaded-force-shading:jar:14.0:compile [INFO] | \- org.apache.flink:flink-table-api-java-bridge_2.12:jar:1.14.5:compile [INFO] +- redis.clients:jedis:jar:5.2.0:compile [INFO] | +- org.apache.commons:commons-pool2:jar:2.12.0:compile [INFO] | +- org.json:json:jar:20240303:compile [INFO] | \- com.google.code.gson:gson:jar:2.11.0:compile [INFO] +- org.apache.httpcomponents:httpclient:jar:4.5.14:compile [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.16:compile [INFO] | +- commons-logging:commons-logging:jar:1.2:compile [INFO] | \- commons-codec:commons-codec:jar:1.11:compile [INFO] +- com.alibaba.fastjson2:fastjson2:jar:2.0.53:compile [INFO] +- org.apache.flink:flink-json:jar:1.20.2:compile [INFO] +- org.projectlombok:lombok:jar:1.18.30:provided [INFO] +- org.apache.hadoop:hadoop-common:jar:3.3.6:compile [INFO] | +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.3.6:compile [INFO] | +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile [INFO] | +- commons-io:commons-io:jar:2.8.0:compile [INFO] | +- commons-net:commons-net:jar:3.9.0:compile [INFO] | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile [INFO] | +- jakarta.activation:jakarta.activation-api:jar:1.2.1:runtime [INFO] | +- org.eclipse.jetty:jetty-server:jar:9.4.51.v20230217:compile [INFO] | | +- org.eclipse.jetty:jetty-http:jar:9.4.51.v20230217:compile [INFO] | | \- org.eclipse.jetty:jetty-io:jar:9.4.51.v20230217:compile [INFO] | +- org.eclipse.jetty:jetty-util:jar:9.4.51.v20230217:compile [INFO] | +- org.eclipse.jetty:jetty-servlet:jar:9.4.51.v20230217:compile [INFO] | | +- org.eclipse.jetty:jetty-security:jar:9.4.51.v20230217:compile [INFO] | | \- org.eclipse.jetty:jetty-util-ajax:jar:9.4.51.v20230217:compile [INFO] | +- org.eclipse.jetty:jetty-webapp:jar:9.4.51.v20230217:compile [INFO] | | \- org.eclipse.jetty:jetty-xml:jar:9.4.51.v20230217:compile [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime [INFO] | +- com.sun.jersey:jersey-core:jar:1.19.4:compile [INFO] | | \- javax.ws.rs:jsr311-api:jar:1.1.1:compile [INFO] | +- com.sun.jersey:jersey-servlet:jar:1.19.4:compile [INFO] | +- com.github.pjfanning:jersey-json:jar:1.20:compile [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile [INFO] | | \- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | | \- javax.xml.stream:stax-api:jar:1.0-2:compile [INFO] | +- com.sun.jersey:jersey-server:jar:1.19.4:compile [INFO] | +- ch.qos.reload4j:reload4j:jar:1.2.22:compile [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.4:compile [INFO] | +- org.apache.commons:commons-configuration2:jar:2.8.0:compile [INFO] | +- org.slf4j:slf4j-reload4j:jar:1.7.36:compile [INFO] | +- org.apache.avro:avro:jar:1.7.7:compile [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile [INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | +- com.google.re2j:re2j:jar:1.1:compile [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.3.6:compile [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile [INFO] | | | \- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile [INFO] | | +- org.apache.curator:curator-framework:jar:5.2.0:compile [INFO] | | \- org.apache.kerby:kerb-simplekdc:jar:1.0.1:compile [INFO] | | +- org.apache.kerby:kerb-client:jar:1.0.1:compile [INFO] | | | +- org.apache.kerby:kerby-config:jar:1.0.1:compile [INFO] | | | +- org.apache.kerby:kerb-common:jar:1.0.1:compile [INFO] | | | | \- org.apache.kerby:kerb-crypto:jar:1.0.1:compile [INFO] | | | +- org.apache.kerby:kerb-util:jar:1.0.1:compile [INFO] | | | \- org.apache.kerby:token-provider:jar:1.0.1:compile [INFO] | | \- org.apache.kerby:kerb-admin:jar:1.0.1:compile [INFO] | | +- org.apache.kerby:kerb-server:jar:1.0.1:compile [INFO] | | | \- org.apache.kerby:kerb-identity:jar:1.0.1:compile [INFO] | | \- org.apache.kerby:kerby-xdr:jar:1.0.1:compile [INFO] | +- com.jcraft:jsch:jar:0.1.55:compile [INFO] | +- org.apache.curator:curator-client:jar:5.2.0:compile [INFO] | +- org.apache.curator:curator-recipes:jar:5.2.0:compile [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.6.3:compile [INFO] | | +- org.apache.zookeeper:zookeeper-jute:jar:3.6.3:compile [INFO] | | +- org.apache.yetus:audience-annotations:jar:0.5.0:compile [INFO] | | +- io.netty:netty-handler:jar:4.1.63.Final:compile [INFO] | | | +- io.netty:netty-common:jar:4.1.63.Final:compile [INFO] | | | +- io.netty:netty-resolver:jar:4.1.63.Final:compile [INFO] | | | +- io.netty:netty-buffer:jar:4.1.63.Final:compile [INFO] | | | +- io.netty:netty-transport:jar:4.1.63.Final:compile [INFO] | | | \- io.netty:netty-codec:jar:4.1.63.Final:compile [INFO] | | +- io.netty:netty-transport-native-epoll:jar:4.1.63.Final:compile [INFO] | | | \- io.netty:netty-transport-native-unix-common:jar:4.1.63.Final:compile [INFO] | | +- org.slf4j:slf4j-log4j12:jar:1.7.25:compile [INFO] | | \- log4j:log4j:jar:1.2.17:compile [INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.2.4:compile [INFO] | +- org.apache.kerby:kerb-core:jar:1.0.1:compile [INFO] | | \- org.apache.kerby:kerby-pkix:jar:1.0.1:compile [INFO] | | +- org.apache.kerby:kerby-asn1:jar:1.0.1:compile [INFO] | | \- org.apache.kerby:kerby-util:jar:1.0.1:compile [INFO] | +- org.codehaus.woodstox:stax2-api:jar:4.2.1:compile [INFO] | +- com.fasterxml.woodstox:woodstox-core:jar:5.4.0:compile [INFO] | +- dnsjava:dnsjava:jar:2.1.7:compile [INFO] | \- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile [INFO] +- org.apache.hadoop:hadoop-hdfs-client:jar:3.3.6:compile [INFO] | +- com.squareup.okhttp3:okhttp:jar:4.9.3:compile [INFO] | | \- com.squareup.okio:okio:jar:2.8.0:compile [INFO] | +- org.jetbrains.kotlin:kotlin-stdlib:jar:1.4.10:compile [INFO] | +- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.4.10:compile [INFO] | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile [INFO] \- org.apache.kafka:kafka-clients:jar:3.6.0:compile [INFO] +- com.github.luben:zstd-jni:jar:1.5.5-1:compile [INFO] \- org.lz4:lz4-java:jar:1.8.0:compile [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 0.779 s [INFO] Finished at: 2025-08-27T18:32:41+08:00 [INFO] ------------------------------------------------------------------------ oscar@oscardeMacBook-Pro bz-sport-realtime % 我的树是这样，那么你说的依赖冲突不存在，为什么还是会报我本地idea上flink任务可以正常消费Kafka，但是上yarn集群就报如下错误： org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:830) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader.<init>(KafkaPartitionSplitReader.java:97) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.connector.kafka.source.KafkaSource.lambda$createReader$1(KafkaSource.java:185) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.createSplitFetcher(SplitFetcherManager.java:259) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.addSplits(SingleThreadFetcherManager.java:148) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.connector.base.source.reader.SourceReaderBase.addSplits(SourceReaderBase.java:315) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.api.operators.SourceOperator.handleAddSplitsEvent(SourceOperator.java:626) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:596) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl.dispatchEventToHandlers(OperatorEventDispatcherImpl.java:72) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.dispatchOperatorEvent(RegularOperatorChain.java:80) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$dispatchOperatorEvent$24(StreamTask.java:1609) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:101) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:414) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:383) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:368) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) [bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [bz-sport-realtime-1.0-SNAPSHOT.jar:?] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_451] Caused by: org.apache.kafka.common.KafkaException: class org.apache.kafka.common.serialization.ByteArrayDeserializer is not an instance of org.apache.kafka.common.serialization.Deserializer at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:405) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:436) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:421) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:709) ~[bz-sport-realtime-1.0-SNAPSHOT.jar:?] ... 26 more
08-28
Flink 任务在本地 IDEA 可正常消费 Kafka，但在 YARN 集群上报 `Failed to construct kafka consumer` 错误，`Caused by: class org.apache.kafka.common.serialization.ByteArrayDeserializer is not an instance ...
Flink 1.18安装 及配置 postgres12 同步到mysql5.7（Flink sql 方式）
山巅
10-17 1506
解决方案：在flink的flink-conf.yaml文件中添加classloader.resolve-order: parent-first 改成parent-first，重启集群即可。flink 下载地址：https://flink.apache.org/downloads/下载 flink 安装包：flink-1.18.1-bin-scala_2.12.tgz。Flink sql 客户端开启： ./sql-client.sh。解决： 在flink -->lib 目录下增加如下jar。
Flink_CDC搭建及简单使用
m0_67392811的博客
08-01 1112
CDC（ChangeDataCapture），在广义的概念上，只要能捕获数据变更的技术，都可以称为CDC。但通常我们说的CDC技术主要面向数据库（包括常见的mysql,Oracle,MongoDB等）的变更，是一种用于捕获数据库中数据变更的技术。目前市面上的CDC技术非常多,常见的主要包括FlinkCDC,DataX,Canal,Sqoop,Kettle,OracleGoldengate,Debezium等。...
flink数据sink入mysql（exactly-once）
weixin_44990574的博客
08-11 4433
flink消费数据sink入mysql，exactly-once事务模式
Flink写入MySQL
鸟人的博客
07-12 3119
在Flink流式数据处理中Sink常用的有Kafka，ES，MySQL，今天来看下，Flink中怎么使用JdbcSink将数据流写入MySQL中POM，在FLINK-2的基础上添加 模拟一些数据，处理后写入MySQL 创建库表SQL 执行代码后，数据库就会有数据了............
mysql8.0升级事项_mysql5X 升级到 mysql8.0 注意事项
weixin_31119221的博客
01-27 393
当安装了Mysql8.0以上版本，搭建SpringBoot环境时启动项目时会遇到以下错误：(1)第一个坑Caused by javax.net.ssl.SSLHandshakeException:java.security.cert.certificateException解决办法pom.xml中的mysql驱动包依赖修改为mysqlmysql-connector-java8.0.11applic...
Flink将数据写入到mysql
FlatTiger的博客
05-29 1176
import org.apache.flink.api.common.functions.MapFunction; import org.apache.flink.connector.jdbc.JdbcConnectionOptions; import org.apache.flink.connector.jdbc.JdbcExecutionOptions; import org.apache.flink.connector.jdbc.JdbcSink; import org.apache.flink.st
flink将数据写入到mysql
congge_study的博客
01-13 8026
flink将数据写入到mysql
flink流处理访问mysql
u012447842的博客
04-10 1万+
业务场景如下： 概述：采集工厂设备的数据。 flink连接emqtt，采集工业物联网的数据，进行流处理，工厂设备数据内没有machID， 需要从mysq的设备信息基础表根据gateMac获取对应的machID。 访问mysql实现思路（一）： flink项目在初始化的时候从mysql获取所有设备的基础信息，所遇问题：如果新增一个设备，flink项目需要重启一次...
Flink 读写MySQL数据（DataStream和Table API）
wank1259162的博客
06-24 9690
Flink提供了基于JDBC的方式，可以将读取到的数据写入到MySQL中；本文通过两种方式将数据下入到MySQL数据库，其他的基于JDBC的数据库类似，另外，Table API方式的Catalog指定为Hive Catalog方式，持久化DDL操作。Maven依赖，包含了Hive Catalog的相关依赖 DataStream方式读写MySQL数据 Table API的方式读写MySQL，其中Flink的Catalog使用Hive Catalog的方式 MySQL中的数据..........
Flink读写系列之-读mysql并写入mysql
热门推荐
A coder of life
10-19 2万+
在Flink文档中，提供connector读取源数据和把处理结果存储到外部系统中。但是没有提供数据库的connector，如果要读写数据库，官网给出了异步IO(Asynchronous I/O)专门用于访问外部数据，详细可看： https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/asyn...
Kafka2.5-＞Flink1.12-＞Mysql8(Jark实验改为DDL形式)
微电子学与固体电子学-俞驰
12-24 956
##############################################实验目的和环境############################################################## 本文是为了在最新的版本上复现[1]，环境如下: 组件 版本 Zookeeper 3.6.0 Flink 1.12 Mysql 8.0.22-0ubuntu0.20.04.2 Kafka 2.5.0 ###########.
大数据时代时序数据库选型指南：为何Apache IoTDB成为物联网场景首
最新发布
小相探索IT世界
01-08 734
时序数据库选型面临物联网、工业互联网等场景下海量数据处理的挑战。本文对比分析了主流时序数据库的核心维度：InfluxDB适合中小规模监控场景，TimescaleDB兼容SQL但性能有限，OpenTSDB扩展性强但架构复杂。重点推荐专为物联网设计的Apache IoTDB，其树状数据模型贴合工业场景，具备千万级写入吞吐量、10:1高压缩比和端边云协同架构。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

IT贫道

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值