Kafka3.0文档 - 1.0 Apache Kafka介绍

Kafka是一个事件实时平台,用于发布和订阅事件流,持久化数据,并处理实时和历史事件。它支持分布式、高扩展、容错和安全的数据处理。事件流可用于实时交易处理、物联网数据收集、用户行为分析等多种场景。Kafka包含生产者、消费者、主题、分区和副本等核心概念,保证数据的可靠性和顺序。其API包括Admin、Producer、Consumer、KafkaStreams和KafkaConnect,方便集成和管理数据流。


在10分钟内了解kafka的一些.

什么是事件流

事件流是数字的等同于人类体内神精系统. 对于始终在线的世界kafka是技术的基基础,业务越来越多的定义软件和自动化,用软件用户也导致有更多的软件.
严格来讲,事件流是实时捕获数据的一个实践,从事件源像 数据库,传感器,手机,云服务和流事件的软件应用.存储事情流为了过后检索.操作,处理,和对事件流的反应等同于回顾,在技术需要时路由实时事件流到不同的目的地.每个事件流确保不断流动和解析数据,导致合适的信息在合适的位置,在合适的时间.

我可以用事件流做什么

事件流被应用在更广泛的例子从大量企业到组织.非常多的例子包括:

  • 实时处理支付和财务交易,库存交易,银行,或保险.
  • 实时跟踪和监控汽车,卡车,舰队,和运载货物,像后勤和自动化工厂.
  • 从IoT装置或其它设备持续捕获或分区传感器数据,像工厂或风力发电场.
  • 收集或立刻反应顾客操作或订单,像零食,旅店和旅游产业和手机应用.
  • 监控医院护理病人和预测在条件下的改变来及时做紧及治疗.
  • 用来连接,存储,并且确保数据被公司不同区域产生.
  • 为了一个服务,数据平台基础,事件驱动架构,和微服务.

Kafka是一个事件实时平台,这是什么意思

kafka结合两个关键功能,这可以实现你使用的例子,为事件实时流,端到端用一个单独经过战斗考研的解决方案:

  1. 发布(写)和订阅(读)事件流,包括连接不断的从其它系统导入/导出你的数据.
  2. 存储事件流长久的有保障的只要你想要
  3. 处理事件流,当他们存在或回顾

所有这些数据被提供在分布的,高扩展,有强性,容错和安全方式.kafka可以部署到 bare-metal硬件,虚拟机,和容器,并且在云中提前定制.你可以选择在kafka自管理环境或使用全管理服务通过不同的供应商.

用简短方式解释Kafka是怎样工作的

kafka是一个分布系统,包含服务端和客户端,通信通过高性能TCP网络协议.可以部署到 bare-metal 硬件,虚拟机,和云环境中的容器中.

服务器端:kafka作为群集中一个或多个服务,可以跨越多个数据中心或云区域.一些来到存储层的服务器,被称为brokers. 其它服务器运行Kafka Connect来不断导入导出数据作为事件流来集成kafka和你已存在的系统,像关系型数据库和kafka集群. 为了让你实现核心任务使用例子,kafka集群是一个高扩展和容错性:如果任何服务器失败,其它服务器会接管他们的工作确保持续任务没有任何数据丢失.

客户端: 允许你写分布式应用程序和微服务来读,写,和并行处理事件流,在可扩容和容错性的方式,甚至在网络有问题或机器出错.kafka包含一些客户端,被扩容通过大量客户端(kafka社区提供).客户端对java,scala可用 包含更高级别的kafka streams库,对go,python,c/c++和其它语言也可以用rest api;

主要概念和术语

一条事件记录发生在实事世界或你的业务中. 在文档中它也叫记录或是消息.当你读或写数据到kafka时,你可以用事件方式来这样做.概念上来说,事件有key,值,和时间戳,和可选的头元数据.下边是事件例子:

  • 事件key:“Alice”
  • 事件值:Made a payment of $200 to Bob"
  • 事件时间戳: “Jun. 25, 2020 at 2:06 p.m.”

生产者是那些客户端发布(写)事件到kafka,并且消费者是那些订阅(读和处理)那些事件.在kafka里,生产者和消费者是解藕的,他们彼此不知道,key设计元素到获得高性能kafka知道.例如,生产者从不需要等待消费者.kafka提供多种保证像事件处理只有一次的能力.

事件是被组织起来并保存在topics.非常简化的说,一个topic非常类似在系统中的一个文件夹,事件像文件夹中文件.一个名为 "payments"的topic.在kafka中的topic总是多个生产者和多个订阅者:一个topic可以有0个,1个或多个生成者来写事件到topic中,同时也有0个,1个或多个消费都来订阅这些事件.topic中的事件,可以被读在需要时.相反你可以定义kafka应在保存你的事件的时间,通过每个topic的配置选项,之后每个过期的事件会被丢弃.Kafka的高效性能对数据大小是恒定的,所以存储长时间的数据是完美的.

Topics是可以被分区的,意味着topic遍布于不同kafka broker上的一些“buckets”. 分布式部署你的数据是对于可扩展来说非常重要,因为允许客户端应用又读又写数据从/到不同的brokers在相同时间.当一个新事件被发布以一个topic,它实际添加一个topic分区.事件带眷相同的key(如,一个客户或车辆ID)被写到相同的分区,并且kafka保护给任何定客户总是读分区事件在精准订单是他们写的.
..
这个例子的topic有p1 - p4分区.两个生产都客户端在生产数据,他们两个相互独立,新的事件通过网络写入到分区.相同key(在图中他们颜色相同)的事件写入到相同分区.注意两个生产者能写数据到相同分区,如果他们的数据合适.

使你的数据具有容错并且高可用,每个topic有副本,甚至跨地区或数据中心,以至于总是有多个brokers有一个副本数据,以防出现什么坏事,你想要做维护在brokers等等.正常生产设置为3个副本,例如,你的数据总是3个副本数据.这些副本执行是在topic分区级别的.

作为初学读物这样介绍足够了.如果你感兴趣,文档中设计部分解释了Kafka各种概念的所有细节.

Kafka API

除了为管理和监控任务的命令行工具外,Kafka有5个为Java和Scala的核心API:

  • Admin API用来管理和检查topics,brokers和其它Kafka对象.
  • Producer API用来发布(写)事件流到一个或更多Kafka topics
  • Consumer API用来订阅(读)一个或多个topics并且处理处理事件流再生成他们.
  • Kafka Streams API 实现流处理应用和微服务.它提供更调味级别功能来处理事件流,包含转换,状态操作像aggregations和joins,windowing,基于事件时间的processing,等等.输入读从一个或更多topics为了生成输出一个或多个topics,高效转换输入流到输出流.
  • Kafka Connect API来构建并运行复用数据导入/导出联接器,消费(读)或生产(写)事件流,并且写到扩展系统和应用,这样他们能集成kafka.例如,连接器到关系性数据库,像PostgreSQL可能获得一组表的每次改变.虽然这样,在实践中,你一般不需要实现你自己的连接器,因为kafka社区已经提供了数百个可用的连接器.

从这里还能去哪

  • 获得kafka实践经验,可以快速开始
  • 为明白kafka更多细节,读文档.你也有你的选择kafka书籍或学术论文.
  • 浏览用户案例来学习在社会里其它用户是怎样从kafka身上获得价值的.
  • 加入本地kafka会议组并且在Kafka峰会讨论,Kafka社会的主要的研讨会.
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob</artifactId> <packaging>pom</packaging> <version>1.0.0-SNAPSHOT</version> <modules> <module>crmp-data-syncjob-common</module> <module>crmp-data-syncjob-dao</module> <module>crmp-data-syncjob-domain</module> <module>crmp-data-syncjob-service</module> <module>crmp-data-syncjob-web</module> </modules> <name>多数据源同步服务</name> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.7.18</version> <relativePath/> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>11</java.version> <!-- 框架版本 --> <flink.version>1.19.3</flink.version> <flink-cdc.version>3.2.0</flink-cdc.version> <debezium.version>1.9.8.Final</debezium.version> <scala.binary.version>2.12</scala.binary.version> <!-- 数据库与中间件依赖 --> <mysql.version>8.0.21</mysql.version> <druid.version>1.2.21</druid.version> <mybatis.version>2.3.1</mybatis.version> <kafka-clients.version>2.8.1</kafka-clients.version> <!-- 工具类与文档 --> <lombok.version>1.18.30</lombok.version> <hutool.version>5.8.6</hutool.version> <commons-lang3.version>3.14.0</commons-lang3.version> <!-- 统一SLF4J版本(与Flink 1.19.3兼容) --> <slf4j.version>1.7.36</slf4j.version> <!-- Infinispan统一版本(避免传递依赖冲突) --> <infinispan.version>13.0.20.Final</infinispan.version> </properties> <!-- 依赖管理:统一锁定版本,优先级:BOM > 显式声明 --> <dependencyManagement> <dependencies> <!-- 1. Debezium BOM:优先锁定所有Debezium子依赖版本(包括传递依赖) --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-bom</artifactId> <version>${debezium.version}</version> <type>pom</type> <scope>import</scope> </dependency> <!-- 2. Spring Cloud 依赖(与Spring Boot 2.7.18兼容) --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>2021.0.8</version> <type>pom</type> <scope>import</scope> </dependency> <!-- 3. 核心依赖版本锁定 --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>${slf4j.version}</version> </dependency> <dependency> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons</artifactId> <version>${infinispan.version}</version> </dependency> <!-- 4. Debezium核心组件(已通过BOM锁定版本,此处仅声明排除规则) --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-core</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <!-- 排除有问题的jdk11专用包 --> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-mysql</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-oracle</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- 5. Flink CDC组件 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>${flink-cdc.version}</version> </dependency> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-oracle-cdc</artifactId> <version>${flink-cdc.version}</version> </dependency> <!-- 6. 子模块版本管理 --> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-common</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-dao</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-domain</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-service</artifactId> <version>${project.version}</version> </dependency> </dependencies> </dependencyManagement> <dependencies> <!-- ========== Spring Boot核心:排除日志冲突 ========== --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <exclusions> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-logging</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-aop</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-log4j2</artifactId> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <optional>true</optional> </dependency> <!-- ========== 测试依赖 ========== --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 工具类 ========== --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>${hutool.version}</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>${commons-lang3.version}</version> </dependency> <dependency> <groupId>commons-codec</groupId> <artifactId>commons-codec</artifactId> <version>1.16.0</version> </dependency> <!-- ========== 数据库依赖 ========== --> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>${mysql.version}</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid-spring-boot-starter</artifactId> <version>${druid.version}</version> <exclusions> <exclusion> <groupId>com.alibaba</groupId> <artifactId>dubbo</artifactId> </exclusion> <exclusion> <groupId>com.googlecode</groupId> <artifactId>hibernate-memcached</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>${mybatis.version}</version> </dependency> <!-- Oracle驱动 --> <dependency> <groupId>com.oracle.database.jdbc</groupId> <artifactId>ojdbc10</artifactId> <version>19.10.0.0</version> <scope>runtime</scope> </dependency> <dependency> <groupId>com.oracle.database.nls</groupId> <artifactId>orai18n</artifactId> <version>19.10.0.0</version> </dependency> <!-- 人大金仓 --> <dependency> <groupId>com.kingbase8.jdbc</groupId> <artifactId>kingbase8</artifactId> <version>8.6.0</version> <scope>runtime</scope> </dependency> <!-- ========== Flink核心依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner_${scala.binary.version}</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-json</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>31.1-jre</version> <scope>provided</scope> </dependency> <!-- ========== CDC连接器(核心修改:排除Infinispan-jdk11依赖) ========== --> <!-- Mysql CDC:排除传递的Debezium和Infinispan问题依赖 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <exclusions> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> </exclusion> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> </exclusion> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <!-- 排除Flink CDC传递的Infinispan问题依赖 --> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- Oracle CDC:排除传递的Debezium和Infinispan问题依赖 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-oracle-cdc</artifactId> <exclusions> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> </exclusion> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- 人大金仓CDC:排除传递的Debezium依赖 --> <dependency> <groupId>com.kingbase</groupId> <artifactId>flink-sql-cdc-connector-kes-v2</artifactId> <version>3.2-SNAPSHOT</version> <exclusions> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 显式引入Debezium核心组件(版本由dependencyManagement锁定) ========== --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-core</artifactId> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-mysql</artifactId> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-oracle</artifactId> </dependency> <!-- ========== Flink补充依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.apache.commons</groupId> <artifactId>commons-math3</artifactId> </exclusion> </exclusions> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-math3</artifactId> <version>3.6.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-base</artifactId> <version>${flink.version}</version> </dependency> <!-- ========== Kafka依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>3.2.0-1.19</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>${kafka-clients.version}</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-api</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-json</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-transforms</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== Doris连接器 ========== --> <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.19</artifactId> <version>25.1.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> <exclusion> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.12.7</version> </dependency> <!-- ========== Hadoop支持 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-hadoop-compatibility_2.12</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>3.3.6</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.apache.yetus</groupId> <artifactId>audience-annotations</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>3.3.6</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.apache.yetus</groupId> <artifactId>audience-annotations</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 其他工具依赖 ========== --> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>2.0.39</version> </dependency> <dependency> <groupId>javax.servlet</groupId> <artifactId>javax.servlet-api</artifactId> <version>3.1.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.github.pagehelper</groupId> <artifactId>pagehelper</artifactId> <version>5.3.2</version> </dependency> <dependency> <groupId>org.json</groupId> <artifactId>json</artifactId> <version>20231013</version> </dependency> </dependencies> <!-- 环境配置 --> <profiles> <profile> <id>dev</id> <activation> <activeByDefault>false</activeByDefault> </activation> <properties> <profileActive>dev</profileActive> </properties> </profile> <profile> <id>prod</id> <activation> <activeByDefault>true</activeByDefault> </activation> <properties> <profileActive>prod</profileActive> </properties> </profile> </profiles> </project>这是pom配置,这是输出 PS D:\kingbaseProject\crmp-data-syncjob> mvn clean dependency:tree -U -Dincludes="io.debezium:*,org.apache.kafka:*" [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] 多数据源同步服务 [pom] [INFO] crmp-data-syncjob-common [jar] [INFO] crmp-data-syncjob-domain [jar] [INFO] crmp-data-syncjob-dao [jar] [INFO] crmp-data-syncjob-service [jar] [INFO] crmp-data-syncjob-web [jar] [INFO] [INFO] -------------------< com.crmp.ecc:crmp-data-syncjob >------------------- [INFO] Building 多数据源同步服务 1.0.0-SNAPSHOT [1/6] [INFO] from pom.xml [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob --- [INFO] com.crmp.ecc:crmp-data-syncjob:pom:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-common >---------------- [INFO] Building crmp-data-syncjob-common 1.0.0-SNAPSHOT [2/6] [INFO] from crmp-data-syncjob-common\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob-common --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-common --- [INFO] com.crmp.ecc:crmp-data-syncjob-common:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-domain >---------------- [INFO] Building crmp-data-syncjob-domain 1.0.0-SNAPSHOT [3/6] [INFO] from crmp-data-syncjob-domain\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob-domain --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-domain --- [INFO] com.crmp.ecc:crmp-data-syncjob-domain:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-dao >----------------- [INFO] Building crmp-data-syncjob-dao 1.0.0-SNAPSHOT [4/6] [INFO] from crmp-data-syncjob-dao\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob-dao --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-dao --- [INFO] com.crmp.ecc:crmp-data-syncjob-dao:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] [INFO] ---------------< com.crmp.ecc:crmp-data-syncjob-service >--------------- [INFO] Building crmp-data-syncjob-service 1.0.0-SNAPSHOT [5/6] [INFO] from crmp-data-syncjob-service\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob-service --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-service --- [INFO] com.crmp.ecc:crmp-data-syncjob-service:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] [INFO] -----------------< com.crmp.ecc:crmp-data-syncjob-web >----------------- [INFO] Building crmp-data-syncjob-web 1.0.0-SNAPSHOT [6/6] [INFO] from crmp-data-syncjob-web\pom.xml [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ crmp-data-syncjob-web --- [INFO] [INFO] --- maven-dependency-plugin:3.3.0:tree (default-cli) @ crmp-data-syncjob-web --- [INFO] com.crmp.ecc:crmp-data-syncjob-web:jar:1.0.0-SNAPSHOT [INFO] +- io.debezium:debezium-core:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-api:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-mysql:jar:1.9.8.Final:compile [INFO] | \- io.debezium:debezium-ddl-parser:jar:1.9.8.Final:compile [INFO] +- io.debezium:debezium-connector-oracle:jar:1.9.8.Final:compile [INFO] +- org.apache.kafka:kafka-clients:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-api:jar:2.8.1:compile [INFO] +- org.apache.kafka:connect-json:jar:2.8.1:compile [INFO] \- org.apache.kafka:connect-transforms:jar:2.8.1:compile [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for 多数据源同步服务 1.0.0-SNAPSHOT: [INFO] [INFO] 多数据源同步服务 ........................................... SUCCESS [ 1.648 s] [INFO] crmp-data-syncjob-common ........................... SUCCESS [ 0.115 s] [INFO] crmp-data-syncjob-domain ........................... SUCCESS [ 0.085 s] [INFO] crmp-data-syncjob-dao .............................. SUCCESS [ 0.131 s] [INFO] crmp-data-syncjob-service .......................... SUCCESS [ 0.113 s] [INFO] crmp-data-syncjob-web .............................. SUCCESS [ 0.121 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.683 s [INFO] Finished at: 2025-11-25T15:00:10+08:00 [INFO] ------------------------------------------------------------------------ ,这还是报错 2025-11-25 14:34:23 java.lang.NoSuchMethodError: &#39;io.debezium.config.Field io.debezium.config.Field.withType(org.apache.kafka.common.config.ConfigDef$Type)&#39; at io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig.<clinit>(HistorizedRelationalDatabaseConnectorConfig.java:48) at io.debezium.relational.RelationalDatabaseConnectorConfig.lambda$new$0(RelationalDatabaseConnectorConfig.java:636) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1810) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.debezium.config.Configuration$5.keys(Configuration.java:1608) at io.debezium.config.Configuration$5.keys(Configuration.java:1605) at io.debezium.config.Configuration.map(Configuration.java:1567) at io.debezium.config.Configuration.subset(Configuration.java:1553) at io.debezium.relational.RelationalDatabaseConnectorConfig.<init>(RelationalDatabaseConnectorConfig.java:638) at io.debezium.connector.kes.KingbaseConnectorConfig.<init>(KingbaseConnectorConfig.java:1024) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:129) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:35) at org.apache.flink.cdc.connectors.base.config.JdbcSourceConfig.getTableFilters(JdbcSourceConfig.java:165) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:219) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:59) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:266) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:248) at org.apache.flink.streaming.api.operators.SourceOperator.handleAddSplitsEvent(SourceOperator.java:625) at org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:595) at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl.dispatchEventToHandlers(OperatorEventDispatcherImpl.java:72) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.dispatchOperatorEvent(RegularOperatorChain.java:80) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$dispatchOperatorEvent$22(StreamTask.java:1548) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:917) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:859) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829) 2025-11-25 14:34:53 java.lang.NoClassDefFoundError: Could not initialize class io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig at io.debezium.relational.RelationalDatabaseConnectorConfig.lambda$new$0(RelationalDatabaseConnectorConfig.java:636) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1810) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.debezium.config.Configuration$5.keys(Configuration.java:1608) at io.debezium.config.Configuration$5.keys(Configuration.java:1605) at io.debezium.config.Configuration.map(Configuration.java:1567) at io.debezium.config.Configuration.subset(Configuration.java:1553) at io.debezium.relational.RelationalDatabaseConnectorConfig.<init>(RelationalDatabaseConnectorConfig.java:638) at io.debezium.connector.kes.KingbaseConnectorConfig.<init>(KingbaseConnectorConfig.java:1024) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:129) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:35) at org.apache.flink.cdc.connectors.base.config.JdbcSourceConfig.getTableFilters(JdbcSourceConfig.java:165) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:219) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:59) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:266) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:248) at org.apache.flink.streaming.api.operators.SourceOperator.handleAddSplitsEvent(SourceOperator.java:625) at org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:595) at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl.dispatchEventToHandlers(OperatorEventDispatcherImpl.java:72) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.dispatchOperatorEvent(RegularOperatorChain.java:80) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$dispatchOperatorEvent$22(StreamTask.java:1548) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:917) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:859) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829)
最新发布
11-26
apache kafka 在数据处理中特删是日志和消息的处理上会有徆多出色的表现.首先弼然推荐的是 kafka 的宏 网 http://kafka.apache.org/。在宏网最值得参考的文章就是 kafka design: http://kafka.apache.org/design.html,要特删重规返篇文章,里面有好多理念都特删好,推荐多读几遍。 在 OSC 的翻译频道有 kafka design 全中文的翻译,翻得挺好的,推荐一下: http://www.oschina.net/translate/kafka-design。kafka 的 wiki 是徆丌错的学习文档: https://cwiki.apache.org/confluence/display/KAFKA/Index 接下来就是一系列文章,文章都是循序渐迕的方式带你了览 kafka: 关亍 kafka 的基本知识,分布式的基础:《分布式消息系统 Kafka 刜步》 kafka 的分布式搭建,quick start:《kafka 分布式环境搭建》 关亍 kafka 的实现细节,返主要就是讲 design 的部分:《细节上》、《细节下》 关亍 kafka 开収环境,scala 环境的搭建:《开収环境搭建》 数据生产者,producer 的用法:《producer 的用法》、《producer 使用注意》 数据消费者,consumer 的用法:《consumer 的用法》 迓有些零碎的,关亍通信段的源码览读:《net 包源码览读》、《broker 配置》 扩展的阅读迓有下面返些: 关亍 kafka 和 jafka 的相关博客,特删好,有徆多问题也都找他览决的,大神一般的存在: http://rockybean.github.com/@rockybean kafka 的 java 化版本 jafka:https://github.com/adyliu/jafka 淘宝的 metaQ:https://github.com/killme2008/Metamorphosis 最近在写的 inforQ,刚开始写,也纯粹是为了读下源码,丌定期更新: https://github.com/ielts0909/inforq
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值