Maven在一个项目中整合Java Spark 与 Scala Spark

最新推荐文章于 2025-06-14 18:08:04 发布

原创最新推荐文章于 2025-06-14 18:08:04 发布 · 3k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#maven #scala #spark #java

Scala 专栏收录该内容

7 篇文章

订阅专栏

本文介绍了如何在一个Maven项目中整合Java和Scala，以及如何添加Spark的相关依赖，包括spark-core_2.11、spark-streaming_2.11、spark-streaming-kafka_2.11和spark-hive_2.11等，确保项目兼容Scala 2.11版本。步骤包括新建Maven项目，添加Scala环境，配置pom.xml文件，并展示最终整合效果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

第一步：

新建一个普通的Maven项目

第二步：

右击项目选择 config --> 选择添加 scala环境，会自动在eclipse/plugins中下载两个文件

org.scala-lang.scala-library.source_2.11.8.v20160304

org.scala-lang.scala-reflect.source_2.11.8.v20160304

第三步：

修改pom.xml文件，导入Java和Scala 开发 Spark需要的jar包

（需要注意的是在第二步添加Scala环境的时候，默认Scala版本是2.11 所以添加Jar也要2.11的版本，否则会提示scala version problem 问题）

<dependencies>
       <dependency>
           <groupId>junit</groupId>
           <artifactId>junit</artifactId>
           <version>3.8.1</version>
           <scope>test</scope>
       </dependency>
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-core_2.11</artifactId>
           <version>1.6.1</version>
           <scope>provided</scope>
       </dependency>
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming_2.11</artifactId>
           <version>1.6.1</version>
       </dependency>

       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming-kafka_2.11</artifactId>
           <version>1.6.1</version>
           <scope>compile</scope>
       </dependency>
       <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-hive_2.11</artifactId>
           <version>1.6.1</version>
           <scope>compile</scope>
       </dependency>

       <dependency>
           <groupId>nl.bitwalker</groupId>
           <artifactId>UserAgentUtils</artifactId>
           <version>1.2.4</version>
           <scope>compile</scope>
       </dependency>
       <dependency>
           <groupId>redis.clients</groupId>
           <artifactId>jedis</artifactId>
           <version>2.8.1</version>
           <scope>compile</scope>
       </dependency>
       <dependency>
           <groupId>com.alibaba</groupId>
           <artifactId>fastjson</artifactId>
           <version>1.2.7</version>
           <scope>compile</scope>
       </dependency>
       <dependency>
           <groupId>org.apache.commons</groupId>
           <artifactId>commons-lang3</artifactId>
           <version>3.3.2</version>
           <scope>compile</scope>
       </dependency>
       <dependency>
           <groupId>org.elasticsearch</groupId>
           <artifactId>elasticsearch-spark_2.11</artifactId>
           <version>2.3.3</version>
           <scope>compile</scope>
       </dependency>
   </dependencies>

   <build>
       <pluginManagement>
           <plugins>
               <plugin>
                   <groupId>org.scala-tools</groupId>
                   <artifactId>maven-scala-plugin</artifactId>
                   <version>2.15.1</version>
                   <executions>
                       <execution>
                           <id>compile-scala</id>
                           <goals>
                               <goal>add-source</goal>
                               <goal>compile</goal>
                           </goals>
                       </execution>
                       <execution>
                           <id>test-compile-scala</id>
                           <goals>
                               <goal>add-source</goal>
                               <goal>testCompile</goal>
                           </goals>
                       </execution>
                   </executions>
               </plugin>
               <plugin>
                   <artifactId>maven-assembly-plugin</artifactId>
                   <version>2.4</version>
                   <configuration>
                       <descriptorRefs>
                           <descriptorRef>jar-with-dependencies</descriptorRef>
                       </descriptorRefs>
                   </configuration>
                   <executions>
                       <execution>
                           <id>make-assembly</id> 
                           <phase>package</phase> 
                           <goals>
                               <goal>single</goal>
                           </goals>
                       </execution>
                   </executions>
               </plugin>

               <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                   <artifactId>maven-compiler-plugin</artifactId>
                   <configuration>
                       <source>1.7</source>
                       <target>1.7</target>
                       <encoding>UTF-8</encoding>
                   </configuration>
               </plugin>
           </plugins>
       </pluginManagement>
   </build>

第四步：

效果如下：