解决：com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat

最新推荐文章于 2024-12-10 13:45:33 发布

原创最新推荐文章于 2024-12-10 13:45:33 发布 · 4.8k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark

错误专栏收录该内容

16 篇文章

订阅专栏

本文详细描述了一个Spark在读取文件时遇到的IllegalAccessError错误，该问题源于特定版本的Hadoop与Spark不兼容。通过将Hadoop版本从2.6.4更改为2.2.0或2.7.2，成功解决了此问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark读取文件的时候报的一个错误：

Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
	at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
	at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
	at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:912)
	at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:910)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.foreach(RDD.scala:910)
	at game.Game01$.main(Game01.scala:17)
	at game.Game01.main(Game01.scala)

pom文件版本

<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
UTF-8
<scala.version>2.10.6</scala.version>
<spark.version>1.6.1</spark.version>
<hadoop.version>2.6.4</hadoop.version>

解决办法：
把hadoop version改成2.2.0就可以了，测了一下2.7.2也可以，所以估计是hadoop的一个坑，2.7.2就修复了。其他版本没试