HashMap源码分析<一>

本文详细剖析了HashMap的工作原理,包括构造方法、装载因子的作用、put方法的实现细节以及resize方法的处理逻辑,尤其关注hash方法对性能的影响及红黑树的应用。

还是从我们使用者的角度一步一步来分析吧。首先我们一般 Map map=new HashMap();
构造方法源码如下。
其实只是初始化了一个装载因子,这个变量干啥用的呢?
loadFactor译为装载因子。装载因子用来衡量HashMap满的程度。loadFactor的默认值为0.75f。计算HashMap的实时装载因子的方法为:size/capacity,而不是占用桶的数量去除以capacity。

 public HashMap() {
        this.loadFactor = DEFAULT_LOAD_FACTOR; // all other fields defaulted
    }

那么我们分析下 map.put(key,value)方法,这个方法代表放入一个键值对。因为HashMap本身就是存储键值对的,key唯一,并且key可以为null。来看下这个方法的源码

public V put(K key, V value) {
    return putVal(hash(key), key, value, false, true);
}

这个hash方法非常关键是关乎性能的。首先理解一个概念
threshold表示当HashMap的size大于threshold时会执行resize操作。
threshold=capacity*loadFactor。可见这个装载因子应该小于1,并且不能太小。

final V putVal(int hash, K key, V value, boolean onlyIfAbsent,
               boolean evict) {         
    Node<K,V>[] tab; Node<K,V> p; int n, i;
    if ((tab = table) == null || (n = tab.length) == 0)
        //第一次操作。初始数组的size。
        n = (tab = resize()).length;
        //相应的key值没有value,那么直接插入,这里是newNode新建。直接调用的是Node构造方法,比较简单。
    if ((p = tab[i = (n - 1) & hash]) == null)
        tab[i] = newNode(hash, key, value, null);
    //这就是所为的发生碰撞,就是同名的key已经有了,这个是不可避免的,也是逻辑所在。 
    //主要逻辑是如果碰撞的对象的hash跟目前对象一样,并且有一些列判断,即时是代表更新原来的对象。   
    else {
        Node<K,V> e; K k;
      //  主要逻辑是如果碰撞的对象的hash跟目前对象一样,并且有一些列判断,即时是代表更新原来的对象。  
        if (p.hash == hash &&
            ((k = p.key) == key || (key != null && key.equals(k))))
            e = p;
      //如果碰撞的对象是一颗数,那么把新的值放进这个树里面。      
        else if (p instanceof TreeNode)
            e = ((TreeNode<K,V>)p).putTreeVal(this, tab, hash, key, value);
      //如果还不是一颗树,只是一个链表,那么要判断是否要进行树化。就是放到这个链表的末尾,如果到末尾的时候,链的数目大于了树化的阀值-1.那么开始树化,链表变成了一个棵树。      
        else {
            for (int binCount = 0; ; ++binCount) {
                if ((e = p.next) == null) {
                    p.next = newNode(hash, key, value, null);
                    if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st
                        treeifyBin(tab, hash);
                    break;
                }
                //如果在链表中找到了自己,那么更新即可。
                if (e.hash == hash &&
                    ((k = e.key) == key || (key != null && key.equals(k))))
                    break;
                p = e;
            }
        }
        //正常应该是进入的。返回了value值。
        if (e != null) { // existing mapping for key
            V oldValue = e.value;
            if (!onlyIfAbsent || oldValue == null)
                e.value = value;
            afterNodeAccess(e);
            return oldValue;
        }
    }
    //修改次数增加。
    ++modCount;
    //判断是否需要重新扩容。
    if (++size > threshold)
        resize();
    afterNodeInsertion(evict);
    return null;
}

介绍的resize方法分析。这个至少包含初始容量扩容和已有容量再扩容。resize针对非链表,链表,树进行了三种不同类型的扩容,因为容量扩大2倍,原先的位置也需要变动。非链表位置是不变的。

  final Node<K,V>[] resize() {
        Node<K,V>[] oldTab = table;
        int oldCap = (oldTab == null) ? 0 : oldTab.length;
        int oldThr = threshold;
        int newCap, newThr = 0;
        if (oldCap > 0) {
            if (oldCap >= MAXIMUM_CAPACITY) {
                threshold = Integer.MAX_VALUE;
                return oldTab;
            }
            else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY &&
                     oldCap >= DEFAULT_INITIAL_CAPACITY)
                newThr = oldThr << 1; // double threshold
        }
        else if (oldThr > 0) // initial capacity was placed in threshold
            newCap = oldThr;
        else {               // zero initial threshold signifies using defaults
            newCap = DEFAULT_INITIAL_CAPACITY;
            newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY);
        }
        if (newThr == 0) {
            float ft = (float)newCap * loadFactor;
            newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ?
                      (int)ft : Integer.MAX_VALUE);
        }
        threshold = newThr;
        @SuppressWarnings({"rawtypes","unchecked"})
            Node<K,V>[] newTab = (Node<K,V>[])new Node[newCap];
        table = newTab;
        if (oldTab != null) {
            for (int j = 0; j < oldCap; ++j) {
                Node<K,V> e;
                if ((e = oldTab[j]) != null) {
                    oldTab[j] = null;
                    if (e.next == null)
//实际上,这里理解成取模运算也没问题。在特定情况下,位运算可以转换成取模运算(当 b = 2^n 时,a % b = a & (b - 1) )。也是因此,HashMap 才将初始长度设置为 16,且扩容只能是以 2 的倍数(2^n)扩容。源码中使用位运算hash & (length - 1) 代替取模运算hash%length,是利用其可以直接对内存数据进行操作,不需要转换成十进制,来提高运算效率。
                        newTab[e.hash & (newCap - 1)] = e;
                    else if (e instanceof TreeNode)
                        ((TreeNode<K,V>)e).split(this, newTab, j, oldCap);
                    else { // preserve order
                    //这个逻辑是把(e.hash & oldCap) == 0
                        Node<K,V> loHead = null, loTail = null;
                        Node<K,V> hiHead = null, hiTail = null;
                        Node<K,V> next;
                        do {
                            next = e.next;
                            if ((e.hash & oldCap) == 0) {
                                if (loTail == null)
                                    loHead = e;
                                else
                                    loTail.next = e;
                                loTail = e;
                            }
                            else {
                                if (hiTail == null)
                                    hiHead = e;
                                else
                                    hiTail.next = e;
                                hiTail = e;
                            }
                        } while ((e = next) != null);
                        if (loTail != null) {
                            loTail.next = null;
                            newTab[j] = loHead;
                        }
                        if (hiTail != null) {
                            hiTail.next = null;
                            newTab[j + oldCap] = hiHead;
                        }
                    }
                }
            }
        }
        return newTab;
    }

先分析到这吧,以后有时间再细化里面的过程,一时半会难以写完,特别是红黑树那块的操作就是一个难点。

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob</artifactId> <packaging>pom</packaging> <version>1.0.0-SNAPSHOT</version> <modules> <module>crmp-data-syncjob-common</module> <module>crmp-data-syncjob-dao</module> <module>crmp-data-syncjob-domain</module> <module>crmp-data-syncjob-service</module> <module>crmp-data-syncjob-web</module> </modules> <name>多数据源同步服务</name> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.7.18</version> <relativePath/> </parent> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>11</java.version> <!-- 框架版本 --> <flink.version>1.19.3</flink.version> <flink-cdc.version>3.2.0</flink-cdc.version> <debezium.version>1.9.8.Final</debezium.version> <scala.binary.version>2.12</scala.binary.version> <!-- 数据库与中间件依赖 --> <mysql.version>8.0.21</mysql.version> <druid.version>1.2.21</druid.version> <mybatis.version>2.3.1</mybatis.version> <kafka-clients.version>2.6.2</kafka-clients.version> <!-- 工具类与文档 --> <lombok.version>1.18.30</lombok.version> <hutool.version>5.8.6</hutool.version> <commons-lang3.version>3.14.0</commons-lang3.version> <!-- 统SLF4J版本(与Flink 1.19.3兼容) --> <slf4j.version>1.7.36</slf4j.version> <!-- Infinispan统版本(避免传递依赖冲突) --> <infinispan.version>13.0.20.Final</infinispan.version> </properties> <!-- 依赖管理:统锁定版本,优先级:BOM > 显式声明 --> <dependencyManagement> <dependencies> <!-- 1. Debezium BOM:优先锁定所有Debezium子依赖版本(包括传递依赖) --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-bom</artifactId> <version>${debezium.version}</version> <type>pom</type> <scope>import</scope> </dependency> <!-- 2. Spring Cloud 依赖(与Spring Boot 2.7.18兼容) --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>2021.0.8</version> <type>pom</type> <scope>import</scope> </dependency> <!-- 3. 核心依赖版本锁定 --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>${slf4j.version}</version> </dependency> <dependency> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons</artifactId> <version>${infinispan.version}</version> </dependency> <!-- 4. Debezium核心组件(已通过BOM锁定版本,此处仅声明排除规则) --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-core</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <!-- 排除有问题的jdk11专用包 --> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-mysql</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-oracle</artifactId> <version>${debezium.version}</version> <exclusions> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- 5. Flink CDC组件 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>${flink-cdc.version}</version> </dependency> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-oracle-cdc</artifactId> <version>${flink-cdc.version}</version> </dependency> <!-- 6. 子模块版本管理 --> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-common</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-dao</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-domain</artifactId> <version>${project.version}</version> </dependency> <dependency> <groupId>com.crmp.ecc</groupId> <artifactId>crmp-data-syncjob-service</artifactId> <version>${project.version}</version> </dependency> </dependencies> </dependencyManagement> <dependencies> <!-- ========== Spring Boot核心:排除日志冲突 ========== --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <exclusions> <exclusion> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-logging</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-aop</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-log4j2</artifactId> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-reload4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <optional>true</optional> </dependency> <!-- ========== 测试依赖 ========== --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> <exclusions> <exclusion> <groupId>org.junit.vintage</groupId> <artifactId>junit-vintage-engine</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 工具类 ========== --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>${hutool.version}</version> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>${commons-lang3.version}</version> </dependency> <dependency> <groupId>commons-codec</groupId> <artifactId>commons-codec</artifactId> <version>1.16.0</version> </dependency> <!-- ========== 数据库依赖 ========== --> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>${mysql.version}</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid-spring-boot-starter</artifactId> <version>${druid.version}</version> <exclusions> <exclusion> <groupId>com.alibaba</groupId> <artifactId>dubbo</artifactId> </exclusion> <exclusion> <groupId>com.googlecode</groupId> <artifactId>hibernate-memcached</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>${mybatis.version}</version> </dependency> <!-- Oracle驱动 --> <dependency> <groupId>com.oracle.database.jdbc</groupId> <artifactId>ojdbc10</artifactId> <version>19.10.0.0</version> <scope>runtime</scope> </dependency> <dependency> <groupId>com.oracle.database.nls</groupId> <artifactId>orai18n</artifactId> <version>19.10.0.0</version> </dependency> <!-- 人大金仓 --> <dependency> <groupId>com.kingbase8.jdbc</groupId> <artifactId>kingbase8</artifactId> <version>8.6.0</version> <scope>runtime</scope> </dependency> <!-- ========== Flink核心依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-api-java-bridge</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-table-planner_${scala.binary.version}</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-json</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>31.1-jre</version> <scope>provided</scope> </dependency> <!-- ========== CDC连接器(核心修改:排除Infinispan-jdk11依赖) ========== --> <!-- Mysql CDC:排除传递的Debezium和Infinispan问题依赖 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <exclusions> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> </exclusion> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> </exclusion> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <!-- 排除Flink CDC传递的Infinispan问题依赖 --> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- Oracle CDC:排除传递的Debezium和Infinispan问题依赖 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-oracle-cdc</artifactId> <exclusions> <exclusion> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> </exclusion> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> </exclusion> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- 人大金仓CDC:排除传递的Debezium依赖 --> <dependency> <groupId>com.kingbase</groupId> <artifactId>flink-sql-cdc-connector-kes-v2</artifactId> <version>3.2-SNAPSHOT</version> <exclusions> <exclusion> <groupId>io.debezium</groupId> <artifactId>*</artifactId> </exclusion> <exclusion> <groupId>org.infinispan</groupId> <artifactId>infinispan-commons-jdk11</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 显式引入Debezium核心组件(版本由dependencyManagement锁定) ========== --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-core</artifactId> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-mysql</artifactId> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-oracle</artifactId> </dependency> <!-- ========== Flink补充依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-core</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java</artifactId> <version>${flink.version}</version> <exclusions> <exclusion> <groupId>org.apache.commons</groupId> <artifactId>commons-math3</artifactId> </exclusion> </exclusions> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-math3</artifactId> <version>3.6.1</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-base</artifactId> <version>${flink.version}</version> </dependency> <!-- ========== Kafka依赖 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka</artifactId> <version>3.2.0-1.19</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>${kafka-clients.version}</version> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-api</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-json</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>connect-transforms</artifactId> <version>${kafka-clients.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== Doris连接器 ========== --> <dependency> <groupId>org.apache.doris</groupId> <artifactId>flink-doris-connector-1.19</artifactId> <version>25.1.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> </exclusion> <exclusion> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.12.7</version> </dependency> <!-- ========== Hadoop支持 ========== --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-hadoop-compatibility_2.12</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>3.3.6</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.apache.yetus</groupId> <artifactId>audience-annotations</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>3.3.6</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.apache.yetus</groupId> <artifactId>audience-annotations</artifactId> </exclusion> </exclusions> </dependency> <!-- ========== 其他工具依赖 ========== --> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>2.0.39</version> </dependency> <dependency> <groupId>javax.servlet</groupId> <artifactId>javax.servlet-api</artifactId> <version>3.1.0</version> <scope>provided</scope> </dependency> <dependency> <groupId>com.github.pagehelper</groupId> <artifactId>pagehelper</artifactId> <version>5.3.2</version> </dependency> <dependency> <groupId>org.json</groupId> <artifactId>json</artifactId> <version>20231013</version> </dependency> </dependencies> <!-- 环境配置 --> <profiles> <profile> <id>dev</id> <activation> <activeByDefault>false</activeByDefault> </activation> <properties> <profileActive>dev</profileActive> </properties> </profile> <profile> <id>prod</id> <activation> <activeByDefault>true</activeByDefault> </activation> <properties> <profileActive>prod</profileActive> </properties> </profile> </profiles> </project>这是配置,这还是报错 2025-11-25 14:34:23 java.lang.NoSuchMethodError: 'io.debezium.config.Field io.debezium.config.Field.withType(org.apache.kafka.common.config.ConfigDef$Type)' at io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig.<clinit>(HistorizedRelationalDatabaseConnectorConfig.java:48) at io.debezium.relational.RelationalDatabaseConnectorConfig.lambda$new$0(RelationalDatabaseConnectorConfig.java:636) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1810) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.debezium.config.Configuration$5.keys(Configuration.java:1608) at io.debezium.config.Configuration$5.keys(Configuration.java:1605) at io.debezium.config.Configuration.map(Configuration.java:1567) at io.debezium.config.Configuration.subset(Configuration.java:1553) at io.debezium.relational.RelationalDatabaseConnectorConfig.<init>(RelationalDatabaseConnectorConfig.java:638) at io.debezium.connector.kes.KingbaseConnectorConfig.<init>(KingbaseConnectorConfig.java:1024) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:129) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:35) at org.apache.flink.cdc.connectors.base.config.JdbcSourceConfig.getTableFilters(JdbcSourceConfig.java:165) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:219) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:59) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:266) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:248) at org.apache.flink.streaming.api.operators.SourceOperator.handleAddSplitsEvent(SourceOperator.java:625) at org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:595) at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl.dispatchEventToHandlers(OperatorEventDispatcherImpl.java:72) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.dispatchOperatorEvent(RegularOperatorChain.java:80) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$dispatchOperatorEvent$22(StreamTask.java:1548) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:917) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:859) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829) 2025-11-25 15:12:28 java.lang.NoClassDefFoundError: Could not initialize class io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig at io.debezium.relational.RelationalDatabaseConnectorConfig.lambda$new$0(RelationalDatabaseConnectorConfig.java:636) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1810) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at io.debezium.config.Configuration$5.keys(Configuration.java:1608) at io.debezium.config.Configuration$5.keys(Configuration.java:1605) at io.debezium.config.Configuration.map(Configuration.java:1567) at io.debezium.config.Configuration.subset(Configuration.java:1553) at io.debezium.relational.RelationalDatabaseConnectorConfig.<init>(RelationalDatabaseConnectorConfig.java:638) at io.debezium.connector.kes.KingbaseConnectorConfig.<init>(KingbaseConnectorConfig.java:1024) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:129) at org.apache.flink.cdc.connectors.kes.source.config.KingbaseSourceConfig.getDbzConnectorConfig(KingbaseSourceConfig.java:35) at org.apache.flink.cdc.connectors.base.config.JdbcSourceConfig.getTableFilters(JdbcSourceConfig.java:165) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:219) at org.apache.flink.cdc.connectors.kes.source.KingbaseDialect.isIncludeDataCollection(KingbaseDialect.java:59) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:266) at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceReader.addSplits(IncrementalSourceReader.java:248) at org.apache.flink.streaming.api.operators.SourceOperator.handleAddSplitsEvent(SourceOperator.java:625) at org.apache.flink.streaming.api.operators.SourceOperator.handleOperatorEvent(SourceOperator.java:595) at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl.dispatchEventToHandlers(OperatorEventDispatcherImpl.java:72) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.dispatchOperatorEvent(RegularOperatorChain.java:80) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$dispatchOperatorEvent$22(StreamTask.java:1548) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:917) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:859) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829) 这问题能不能解决了?
最新发布
11-26
<think> 根据用户提供的错误信息和项目配置,核心问题在于Debezium和Kafka客户端版本不兼容。Debezium 1.9.8.Final需要Kafka客户端2.6.x版本,但项目中使用了2.8.1版本,导致类初始化失败。 ### 解决方案 #### 步骤1:降低Kafka客户端版本 在Maven的`<properties>`中将Kafka客户端版本降级到2.6.2(与Debezium 1.9.8兼容): ```xml <properties> <kafka-clients.version>2.6.2</kafka-clients.version> </properties> ``` #### 步骤2:确保所有模块使用统版本 在父POM的`<dependencyManagement>`中显式锁定Kafka客户端版本: ```xml <dependencyManagement> <dependencies> <!-- 锁定Kafka客户端版本 --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>${kafka-clients.version}</version> </dependency> </dependencies> </dependencyManagement> ``` #### 步骤3:排除冲突传递依赖 在Flink CDC连接器依赖中排除旧版Kafka客户端: ```xml <dependencies> <!-- Flink CDC连接器 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>${flink-cdc.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <!-- 其他Flink CDC连接器同样处理 --> </dependencies> ``` ### 验证依赖树 执行以下命令确认Kafka客户端版本已统: ```bash mvn clean dependency:tree -Dincludes="org.apache.kafka:kafka-clients" ``` 预期输出应显示所有模块使用`kafka-clients:2.6.2`。 ### 备选方案:升级Debezium(风险较高) 若必须使用Kafka 2.8.x,需升级Debezium至2.x版本(如2.3.0.Final): ```xml <properties> <debezium.version>2.3.0.Final</debezium.version> <kafka-clients.version>3.2.4</kafka-clients.version> <!-- Debezium 2.x需要Kafka 3.x --> </properties> ``` **注意**:此方案需同步升级Flink CDC至3.2.0+版本,且需全面测试与Flink 1.19.3的兼容性。 ### 错误原因分析 - **`NoSuchMethodError`**:Debezium 1.9.8中的`Field.withType()`方法签名与Kafka 2.8.1的`ConfigDef$Type`类不匹配。 - **`NoClassDefFoundError`**:因上述方法缺失导致`HistorizedRelationalDatabaseConnectorConfig`类初始化失败。 ### 引用说明 1. 类似依赖冲突问题可通过调整版本解决[^1] 2. 镜像仓库通常不影响类加载问题[^2] §§相关问题§§ 1. 如何检查Maven依赖冲突? 2. Debezium与Kafka的版本兼容性规则是什么? 3. Flink CDC如何选择兼容的Debezium版本? </think> 根据您的错误信息和项目依赖配置,核心问题是 **Debezium 1.9.8.Final 与 Kafka-clients 2.8.1 版本不兼容**。以下是完整解决方案: ### 解决方案(首选) #### 步骤1:降低 Kafka-clients 版本 ```xml <!-- 修改 properties 部分 --> <properties> <!-- 将 Kafka 版本降级至与 Debezium 1.9.8 兼容的版本 --> <kafka-clients.version>2.6.2</kafka-clients.version> </properties> ``` #### 步骤2:在所有依赖中排除冲突的 Kafka-clients ```xml <dependencies> <!-- Flink CDC 连接器 --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <!-- 人大金仓 CDC 连接器 --> <dependency> <groupId>com.kingbase</groupId> <artifactId>flink-sql-cdc-connector-kes-v2</artifactId> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> <!-- 显式引入兼容版本的 Kafka-clients --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>${kafka-clients.version}</version> </dependency> </dependencies> ``` ### 备选方案(升级 Debezium) ```xml <properties> <!-- 升级 Debezium 和 Kafka --> <debezium.version>2.3.0.Final</debezium.version> <kafka-clients.version>3.2.4</kafka-clients.version> <flink-cdc.version>3.2.0</flink-cdc.version> </properties> <dependencies> <!-- 显式升级 Debezium --> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-core</artifactId> <version>${debezium.version}</version> </dependency> <dependency> <groupId>io.debezium</groupId> <artifactId>debezium-connector-mysql</artifactId> <version>${debezium.version}</version> </dependency> <!-- 升级 Flink CDC --> <dependency> <groupId>com.ververica</groupId> <artifactId>flink-connector-mysql-cdc</artifactId> <version>${flink-cdc.version}</version> <exclusions> <exclusion> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> </exclusion> </exclusions> </dependency> </dependencies> ``` ### 错误原因深度分析 1. **类初始化失败(`NoClassDefFoundError`)** `HistorizedRelationalDatabaseConnectorConfig` 静态初始化时调用 `Field.withType()` 方法失败 ```java // Debezium 1.9.8 源码片段 public static final Field SERVER_NAME = Field.create("database.server.name") .withType(ConfigDef.Type.STRING) // 此方法签名在kafka-clients 2.8.1中已改变 .withImportance(ConfigDef.Importance.HIGH); ``` 2. **版本兼容矩阵** | 组件 | 兼容版本 | 冲突版本 | |----------------|------------------------|----------------| | Debezium 1.9.x | Kafka-clients ≤ 2.6.x | Kafka 2.8.1 | | Debezium 2.x | Kafka-clients ≥ 3.x | Kafka < 3.x | ### 验证步骤 1. **检查依赖树** ```bash mvn clean dependency:tree -Dincludes="io.debezium:*,org.apache.kafka:*" ``` 确认输出中只有 **个** Kafka-clients 版本 2. **字节码验证工具** ```java public class DebeziumCompatibilityChecker { public static void main(String[] args) { try { Class.forName("io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig"); ConfigDef.Type.STRING.getClass(); // Kafka 核心类 System.out.println("✅ 版本兼容性验证通过"); } catch (Exception e) { System.err.println("❌ 类加载失败: " + e.getMessage()); } } } ``` ### 附加优化 ```xml <!-- 在 build 插件中添加字节码校验 --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-enforcer-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <id>enforce-versions</id> <goals> <goal>enforce</goal> </goals> <configuration> <rules> <bannedDependencies> <!-- 禁止引入不兼容的 Kafka 版本 --> <excludes> <exclude>org.apache.kafka:kafka-clients:(,2.6.0],[2.7,)</exclude> </excludes> </bannedDependencies> </rules> </configuration> </execution> </executions> </plugin> ``` ### 结果预期 执行修改后: 1. `NoSuchMethodError` 和 `NoClassDefFoundError` 错误将消失 2. Debezium 连接器能正常初始化 3. 人大金仓 CDC 连接器可加载历史表配置 > **首选降级方案**,因为: > 1. 与现有 Flink 1.19.3 兼容性更好 > 2. 不需要修改业务代码 > 3. 风险更低(仅降低个依赖版本)
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值