当然可以！以下是 **完全基于你当前环境
和版本要求**（
Scala 2.12.11,
Spark 3
.1
.1, Hadoop 3
.2.4,
JDK 8）重新整理的、**可直接使用且能正常运行的 `
pom.xml` 文件**。
---
### ✅ 修复目标
- ✅ 保留所有原始
依赖版本不变
- ✅ 解决 `NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder`
- ✅ 消除 SLF4J 多绑定冲突
- ✅ 添加日志实现（`reload4j`）
- ✅ 正确打
包为可执行 fat jar
- ✅ 支持本地运行
Spark + Hive
- ✅ 避免 Guava、log4j 等常见
依赖冲突
---
### ✅ 修改后的完整 `
pom.xml`
```
xml
<?
xml version="1
.0" encoding="UTF-8"?>
<project
xmlns="http://
maven.apache
.org/
POM/4
.0
.0"
xmlns:xsi="http://www
.w3
.org/
2001/
XMLSchema-instance"
xsi:schemaLocation="http://
maven.apache
.org/
POM/4
.0
.0
http://
maven.apache
.org/xsd/
maven-4
.0
.0
.xsd">
<modelVersion>4
.0
.0</modelVersion>
<groupId>org
.example</groupId>
<artifactId>
scala_tutorial</artifactId>
<version>1
.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<
maven.compiler
.source>8</
maven.compiler
.source>
<
maven.compiler
.target>8</
maven.compiler
.target>
<project
.build
.sourceEncoding>UTF-8</project
.build
.sourceEncoding>
<
scala.version>
2.12.11</
scala.version>
<
spark.version>3
.1
.1</
spark.version>
<hadoop
.version>3
.2.4</hadoop
.version>
<main
.class>MySQLToHiveIncremental
2</main
.class> <!-- 替换为你主类的全名 -->
</properties>
<dependencies>
<!-- MySQL Connector -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-
java</artifactId>
<version>8
.0
.33</version>
</dependency>
<!--
Scala Library -->
<dependency>
<groupId>org
.scala-lang</groupId>
<artifactId>
scala-library</artifactId>
<version>${
scala.version}</version>
</dependency>
<!--
Spark Core -->
<dependency>
<groupId>org
.apache
.spark</groupId>
<artifactId>
spark-core_
2.12</artifactId>
<version>${
spark.version}</version>
<exclusions>
<!-- 排除旧版 log4j
和 slf4j-log4j
12，避免冲突 -->
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org
.slf4j</groupId>
<artifactId>slf4j-log4j
12</artifactId>
</exclusion>
<!-- 统一 Guava
版本 -->
<exclusion>
<groupId>com
.google
.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--
Spark SQL -->
<dependency>
<groupId>org
.apache
.spark</groupId>
<artifactId>
spark-sql_
2.12</artifactId>
<version>${
spark.version}</version>
<exclusions>
<!-- 防止引入多余的日志绑定 -->
<exclusion>
<groupId>org
.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<!--
Spark Hive Support -->
<dependency>
<groupId>org
.apache
.spark</groupId>
<artifactId>
spark-hive_
2.12</artifactId>
<version>${
spark.version}</version>
</dependency>
<!-- Hadoop Client -->
<dependency>
<groupId>org
.apache
.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop
.version}</version>
<exclusions>
<!-- 排除 hadoop 自带的 log4j
和 guava -->
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org
.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
<exclusion>
<groupId>com
.google
.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- ✅ 必须添加：SLF4J 日志实现（使用 reload4j） -->
<dependency>
<groupId>org
.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1
.7
.35</version>
</dependency>
<dependency>
<groupId>ch
.qos
.reload4j</groupId>
<artifactId>reload4j</artifactId>
<version>1
.2.18
.3</version>
</dependency>
<dependency>
<groupId>org
.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
<version>1
.7
.35</version>
</dependency>
<!-- ✅ 统一 Guava
版本，避免 NoSuchMethodError -->
<dependency>
<groupId>com
.google
.guava</groupId>
<artifactId>guava</artifactId>
<version>
27
.0-jre</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/
scala</sourceDirectory>
<testSourceDirectory>src/test/
scala</testSourceDirectory>
<plugins>
<!--
Scala Compiler Plugin -->
<plugin>
<groupId>net
.alchim31
.maven</groupId>
<artifactId>
scala-
maven-plugin</artifactId>
<version>4
.8
.1</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<
scalaVersion>${
scala.version}</
scalaVersion>
</configuration>
</plugin>
<!--
Maven Compiler Plugin -->
<plugin>
<groupId>org
.apache
.maven.plugins</groupId>
<artifactId>
maven-compiler-plugin</artifactId>
<version>3
.11
.0</version>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<!--
Maven Assembly Plugin: 打
包成可执行 fat jar -->
<plugin>
<groupId>org
.apache
.maven.plugins</groupId>
<artifactId>
maven-assembly-plugin</artifactId>
<version>3
.6
.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>${main
.class}</mainClass>
</manifest>
</archive>
<finalName>
scala_tutorial</finalName>
<appendAssemblyId>false</appendAssemblyId>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
```
---
### ✅ 使用说明
#### 1
. 创建日志
配置文件
在 `src/main/resources/log4j
.properties` 中创建：
```properties
log4j
.rootLogger=INFO, console
log4j
.appender
.console=org
.apache
.log4j
.ConsoleAppender
log4j
.appender
.console
.target=System
.err
log4j
.appender
.console
.layout=org
.apache
.log4j
.PatternLayout
log4j
.appender
.console
.layout
.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %-5p %c{1}: %m%n
# 减少第三方库的日志噪音
log4j
.logger
.org
.eclipse
.jetty=WARN
log4j
.logger
.org
.apache
.hadoop=WARN
log4j
.logger
.org
.sparkproject
.jetty=WARN
```
####
2. 编译打
包
```bash
mvn clean package
```
输出：`target/
scala_tutorial
.jar`
#### 3
. 运行程序
确保你的主类中有 `
.master("local[*]")`：
```
scala
val
spark =
SparkSession
.builder()
.appName("MySQL to Hive")
.master("local[*]") // 必须设置
.enableHiveSupport()
.getOrCreate()
```
然后运行：
```bash
java -cp target/
scala_tutorial
.jar MySQLToHiveIncremental
2
```
或使用 `
spark-submit`（推荐生产）：
```bash
spark-submit --class MySQLToHiveIncremental
2 target/
scala_tutorial
.jar
```
---
### ✅ 成功标志
你会看到：
```
INFO
SparkContext: Running
Spark version 3
.1
.1
INFO
SparkUI: Started at http://xxx:4040
...
```
并且不再有 `NoClassDefFoundError` 或 `Multiple bindings` 警告。
---
得到如下结果，我们点开Spark-core这个，就可以看到如何在pom.xml中写Spark-core的配置了，如果需要你也可以看到Spark SQL、Spark Streaming等都有，可以自行查看
进去后下滑会看到相关的Spark版本和Scala版本信息，我们以Spark3.4.1版本举例，它兼容的Scala版本是Scala2.12.x和Scala2.13.x版本，这里就解决了如何知道Spark版本与Scala版本是否兼容的问题
扫一扫
1702
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
