flink 报错ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer

最新推荐文章于 2025-03-21 14:24:40 发布

原创

最新推荐文章于 2025-03-21 14:24:40 发布

· 1.1k 阅读

5 ·

版权

文章标签：

#flink #kafka

文章讲述了在使用ApacheFlink将socket流数据写入Kafka时遇到的KafkaProducer构造失败问题，原因是由于类加载器顺序导致的dependencyconflict。解决方案是修改flink-conf.yml中的classloader.resolve-order参数为parent-first。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

问题：
代码逻辑为从socket流中读取数据写入kafka

        Configuration conf = new Configuration();
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(conf);
        env.setParallelism(4);
        env.enableCheckpointing(2000, CheckpointingMode.EXACTLY_ONCE);

        DataStreamSource<String> source = env.socketTextStream("172.18.26.53", 7777);

        KafkaSink<String> sink = KafkaSink.<String>builder()
                .setBootstrapServers("172.18.26.218:9092")
                .setRecordSerializer(
                        KafkaRecordSerializationSchema.<String>builder()
                                .setTopic("test-topic")
                                .setValueSerializationSchema(new SimpleStringSchema())
                                .build())
                .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
                .setTransactionalIdPrefix("cz-")
                .setProperty(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 1 * 60 * 1000 + "")
                .build();
        source.sinkTo(sink);

        env.execute();

命令行bin/flink run-application -t yarn-application -Dyarn.provided.lib.dirs="hdfs://nameservice1/flink-dist" -c com.hex.cz.CZDemo examples/FlinkTutorial-1.17-1.0-SNAPSHOT.jar提交到yarn集群后报错

原因：
查看flink作业日志yarn logs -applicationId application_1706238034141_6936

org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:439) ~[FlinkTutorial-1.17-1.0-SNAPSHOT.jar:?]
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer