spark streaming error -- java.lang.AbstractMethodError

原创

于 2019-07-16 18:12:00 发布 · 1.2k 阅读

CC 4.0 BY-SA版权

博客内容简述：在升级Spark on Yarn集群到2.4版本后，遇到java.lang.AbstractMethodError异常。该问题是由于项目中使用的Spark版本（2.2）与集群版本不匹配导致。解决方案是将项目中的Spark版本更新到与集群相同的2.4版本。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark streaming程序提交出现如下异常：

java.lang.AbstractMethodError
	at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
	at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
	at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
	at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
	at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:255)
	at org.apache.sp