Kafka:ZK+Kafka+Spark Streaming集群环境搭建(二十三)Structured Streaming遇到问题:Set(TopicName-0) are gone. Some da...

本文详细解析了StructuredStreaming在Kafka环境下,由于Topic名称变更及Checkpoint未及时更新导致的数据丢失警告与异常处理过程。强调了正确配置和管理Checkpoint的重要性,以避免数据处理中的潜在损失。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

事情经过:之前该topic(M_A)已经存在,而且正常使用structured streaming消费了一段时间,后来删除了topic(M_A),重新创建了topic(M-A),程序使用新创建的topic(M-A)进行实时统计操作,使用structured streaming执行过程中抛出了一下异常:

18/08/24 10:20:42 INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
18/08/24 10:20:42 INFO utils.AppInfoParser: Kafka commitId : unknown
18/08/24 10:20:42 INFO internals.AbstractCoordinator: Discovered coordinator vmxx.xx.xx.xx.com.cn:9092 (id: 2147483417 rack: null) for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0.
18/08/24 10:20:42 INFO internals.ConsumerCoordinator: Revoking previously assigned partitions [] for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0
18/08/24 10:20:42 INFO internals.AbstractCoordinator: (Re-)joining group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0
18/08/24 10:20:45 INFO internals.AbstractCoordinator: Successfully joined group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0 with generation 1
18/08/24 10:20:45 INFO internals.ConsumerCoordinator: Setting newly assigned partitions [M-A-0] for group spark-kafka-source-165bc430-5cbc-4cfc-8327-9af01fd02fcc-616947503-driver-0
18/08/24 10:20:46 WARN kafka010.KafkaSource: Set(M_A-0) are gone. Some data may have been missed. 
Some data may have been lost because they are not available in Kafka any more; either the
data was aged out by Kafka or the topic may have been deleted before all the data in the
topic was processed. If you want your streaming query to fail on such cases, set the source
option "failOnDataLoss" to "true".

错误原因,在structured streaming编程时,使用checkpoint(checkpointt中添加topicname.replace("-","").replace("_","")),此时忘记了删除checkpoint,因此导致操作。

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值