问题:
代码逻辑为从socket流中读取数据写入kafka
Configuration conf = new Configuration();
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(conf);
env.setParallelism(4);
env.enableCheckpointing(2000, CheckpointingMode.EXACTLY_ONCE);
DataStreamSource<String> source = env.socketTextStream("172.18.26.53", 7777);
KafkaSink<String> sink = KafkaSink.<String>builder()
.setBootstrapServers("172.18.26.218:9092")
.setRecordSerializer(
KafkaRecordSerializationSchema.<String>builder()
.setTopic("test-topic")
.setValueSerializationSchema(new SimpleStringSchema())
.build())
.setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
.setTransactionalIdPrefix("cz-")
.setProperty(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 1 * 60 * 1000 + "")
.build();
source.sinkTo(sink);
env.execute();
命令行bin/flink run-application -t yarn-application -Dyarn.provided.lib.dirs="hdfs://nameservice1/flink-dist" -c com.hex.cz.CZDemo examples/FlinkTutorial-1.17-1.0-SNAPSHOT.jar
提交到yarn集群后报错
原因:
查看flink作业日志yarn logs -applicationId application_1706238034141_6936
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:439) ~[FlinkTutorial-1.17-1.0-SNAPSHOT.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer