Spark:常用JAR包集成（pom.xml）

最新推荐文章于 2022-02-11 09:25:10 发布

drl_blogs

最新推荐文章于 2022-02-11 09:25:10 发布

阅读量716

点赞数

CC 4.0 BY-SA版权

分类专栏： # Spark

本文链接：https://blog.youkuaiyun.com/drl_blogs/article/details/92851719

本文详细介绍了在Spark项目中如何正确集成和管理关键的JAR包，通过调整`pom.xml`文件，确保项目的依赖完整，避免运行时出现类找不到或版本冲突等问题。了解并掌握这些步骤对于成功构建和运行Spark应用程序至关重要。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <scala.version>2.11.8</scala.version>
        <spark.version>2.2.2</spark.version>
        <hadoop.version>2.7.3</hadoop.version>
        <mysql.version>5.1.47</mysql.version>
        <encoding>UTF-8</encoding>
    </properties>
    <dependencies>
        <!-- 导入scala的依赖 -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <!-- 导入spark的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 导入spark-sql的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 指定hadoop-client API的版本 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector -->
        <dependency>
            <groupId>org.mongodb.spark</groupId>
            <artifactId>mongo-spark-connector_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>

        <!-- 导入spark-hive的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- 导入spark-streaming的依赖 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>${mysql.version}</