java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.HiveException

最新推荐文章于 2025-03-18 11:50:16 发布

青山流水在深谷

最新推荐文章于 2025-03-18 11:50:16 发布

阅读量4.3k

点赞数 1

分类专栏： spark 文章标签： spark

本文链接：https://blog.youkuaiyun.com/dymkkj/article/details/117926376

版权

spark 专栏收录该内容

34 篇文章

订阅专栏

博客讲述了在使用Spark读取Hive数据时遇到的ClassNotFoundException问题，原因是Spark依赖的Scala版本与实际使用的不一致。通过检查并修改项目的Scala和Spark依赖版本，将Scala从2.12.4改为2.11.8，Spark从2.4.2改为2.4.0，解决了问题。最后，成功运行了Spark提交的作业。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

代码

测试代码

import org.apache.spark.sql.SparkSession

object SparkHive {
  def main(args: Array[String]): Unit = {

    val spark = SparkSession
      .builder()
      .master("local[1]")
      .appName("Spark Hive Example")
      //.config("spark.sql.warehouse.dir", warehouseLocation)
      .enableHiveSupport()
      .getOrCreate()

    import spark.implicits._
    import spark.sql

    val date = "20201121";
    //hive表读取
    sql(s"select * from ms_access_partition  where ms_access_partition.partdate = '$date'").show(3)

  }

}

错误

spark 读取hive时，出现如下错误
java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.HiveException

检查

依赖检查

<properties>
    <scala.version>2.11.8</scala.version>
    <spotless.version>1.31.3</spotless.version>
    <spark.version>2.4.0</spark.version>
  </properties>
 
<dependencies>
  <!-- spark-core -->
 <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

   <!-- spark-sql -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

   <!-- spark-read-hive -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>


    <!-- hive-jdbc -->
    <dependency>
      <groupId>org.apache.hive</groupId>
      <artifactId>hive-jdbc</artifactId>
      <version>${hive.version}</version>
    </dependency>
    <!--  Hadoop client-->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.6.5</version>
    </dependency>
  </dependencies>

连接设置

1.检查 resource下的core-site.xml,hdfs-site.xml,hive-site.xml配置是否正确
2.检查代码问题

本次原因

代码依赖错误，spark依赖的scala时2.12.4，spark时spark-*-2.12,导致问题

 <properties>
    <scala.version>2.12.4</scala.version>
    <spotless.version>1.31.3</spotless.version>
    <spark.version>2.4.2</spark.version>
  </properties>

<dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.specs</groupId>
      <artifactId>specs</artifactId>
      <version>1.2.5</version>
      <scope>test</scope>
    </dependency>

  <dependencies>
	<!-- spark-core -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.12</artifactId>
      <version>${spark.version}</version>
    </dependency>

	<!-- spark-sql -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.12</artifactId>
      <version>${spark.version}</version>
    </dependency>

  	<!-- spark-hive -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.12</artifactId>
      <version>${spark.version}</version>
    </dependency>
    
    <!-- hive-jdbc -->
    <dependency>
      <groupId>org.apache.hive</groupId>
      <artifactId>hive-jdbc</artifactId>
      <version>2.1.1</version>
    </dependency>
    
      <!-- hadoop-client -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.6.5</version>
    </dependency>
  </dependencies>

解决

修改scala,spark依赖配置

<properties>
    <scala.version>2.11.8</scala.version>
    <spotless.version>1.31.3</spotless.version>
    <spark.version>2.4.0</spark.version>
  </properties>
 
<dependencies>
  <!-- spark-core -->
 <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

   <!-- spark-sql -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

   <!-- spark-read-hive -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>


    <!-- hive-jdbc -->
    <dependency>
      <groupId>org.apache.hive</groupId>
      <artifactId>hive-jdbc</artifactId>
      <version>${hive.version}</version>
    </dependency>
    <!--  Hadoop client-->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.6.5</version>
    </dependency>
  </dependencies>

注意，spark-2.4.2有个问题，是唯一一个spark 2.4.x中用 scala 2.12.x编译的，避免使用该版本

修改后运行

spark-submit \
--class com.wacai.cn.SparkHive \
 SparkSql-1.0-jar-with-dependencies.jar