Flinkcdc同步mysql数据变化报错Failed to execute sql 、stream classdesc serialVersionUID = 8983367024417715979

先看报错

报错信息:截取了部分关键信息报错

Exception in thread "main" org.apache.flink.table.api.TableException: Failed to execute sql
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1060)
	Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	Caused by: java.io.InvalidClassException: com.fasterxml.jackson.databind.cfg.MapperConfigBase; local class incompatible: stream classdesc serialVersionUID = 8983367024417715979, local class serialVersionUID = 6477281625127050473

搜了一下大概是因为flink中的jackson版本依赖和我程序的版本依赖冲突，由于这是一个demo项目，jar包路径都是硬编码。部署到服务器上的话也会因为高低版本的覆盖问题找不到jar包：

最开始的解决方案是（修改硬编码中jar的版本）

最后通过将硬编码传递的jar包改写成动态获取服务器上的jar

解决！

最后附上源码

public class CdcSyncServiceImpl implements CdcSyncService {
    
    private static File libs;
    /*
      在项目中构建专门地导出文件夹
     */
    static {
        URL resource = Thread.currentThread().getContextClassLoader().getResource("");
        if (resource == null) {
            resource = Thread.currentThread().getContextClassLoader().getResource("/");
        }
        if (resource != null) {
          String  CLASS_PATH = new File(resource.getPath()).getPath();
            libs = new File(new File(CLASS_PATH).getParent() + "/libs"+"/bdp-flink-cdc");
        }
    }

    // 动态加载jar
    private String[] getJarFiles() {
        if (libs == null || !libs.exists()) {
            throw new RuntimeException("Libs directory not found");
        }
        File[] files = libs.listFiles((dir, name) -> name.endsWith(".jar"));
        if (files == null || files.length == 0) {
            throw new RuntimeException("No jar files found in libs directory");
        }
        return java.util.Arrays.stream(files)
                .map(file -> libs + "/" + file.getName())
                .toArray(String[]::new);
    }

    @Override
    public void startSync(DataSourceConfig sourceConfig, DataSourceConfig targetConfig) throws Exception {
        StreamExecutionEnvironment env = null;
        String id = java.util.UUID.randomUUID().toString();
        // 创建配置对象
        Configuration config = new Configuration();
        // 指定 Web UI 端口
        config.setInteger(RestOptions.PORT, 8082);
        // 创建 Flink 流执行环境

        //本地环境
//        env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config);
        env = StreamExecutionEnvironment.createRemoteEnvironment(
                "yourhost", 
                8081,
                getJarFiles()  // 使用动态获取的jar包路径
        );
        env.enableCheckpointing(3000L).setParallelism(2);
        EnvironmentSettings settings = EnvironmentSettings
                .newInstance()
                .inStreamingMode()
                .build();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, settings);

        // 创建源表和目标表
        tableEnv.executeSql(generateSourceDDL(sourceConfig));
        tableEnv.executeSql(generateSinkDDL(targetConfig));

        // 执行同步
        tableEnv.getConfig().set("pipeline.name", "Flink CDC - Table Sync"+id);
        tableEnv.executeSql(String.format("insert into %s select * from %s;",
                targetConfig.getTableName(), sourceConfig.getTableName()));

    }
        private String generateSourceDDL(DataSourceConfig config) {
        return String.format("CREATE TABLE %s (\n" +
                        "  id STRING,\n" +
                        "  name STRING,\n" +
                        "  PRIMARY KEY (id) NOT ENFORCED\n" +
                        ") WITH (\n" +
                        "   'connector' = 'mysql-cdc',\n" +
                        "   'hostname' = '%s',\n" +
                        "   'port' = '%s',\n" +
                        "   'username' = '%s',\n" +
                        "   'password' = '%s',\n" +
                        "   'database-name' = '%s',\n" +
                        "   'table-name' = '%s'\n" +
                        ");",
                config.getTableName(),
                config.getHostname(),
                config.getPort(),
                config.getUsername(),
                config.getPassword(),
                config.getDatabase(),
                config.getTableName());
    }

    private String generateSinkDDL(DataSourceConfig config) {
        return String.format("CREATE TABLE %s (\n" +
                        "  id STRING,\n" +
                        "  name STRING,\n" +
                        "  PRIMARY KEY (id) NOT ENFORCED\n" +
                        ") WITH (\n" +
                        "   'connector' = 'jdbc',\n" +
                        "   'url' = 'jdbc:mysql://%s:%s/%s',\n" +
                        "   'driver' = 'com.mysql.cj.jdbc.Driver',\n" +
                        "   'username' = '%s',\n" +
                        "   'password' = '%s',\n" +
                        "   'table-name' = '%s'\n" +
                        ");",
                config.getTableName(),
                config.getHostname(),
                config.getPort(),
                config.getDatabase(),
                config.getUsername(),
                config.getPassword(),
                config.getTableName());

    }

}

总结：原本的硬编码问题应该是最开始要解决的，后来打算最后在优化这个硬编码传入jar包，所以依赖冲突的问题一直得不到解决，一直在Pom中去排除jackson依赖,发现并没有效果，一直越走越远。最后将jar包的传递方式变成动态获取，解决了所有的连锁问题。