spark读取Excel文件：

最新推荐文章于 2025-04-21 10:33:55 发布

熬夜的小王

最新推荐文章于 2025-04-21 10:33:55 发布

阅读量2.7k

点赞数

分类专栏：大数据专栏文章标签： spark 大数据 hadoop

本文链接：https://blog.youkuaiyun.com/weixin_44947939/article/details/125429751

版权

大数据专栏专栏收录该内容

10 篇文章

订阅专栏

该博客展示了如何在Scala中配置Maven项目，引入必要的依赖，特别是`spark-excel`库，以便使用Apache Spark从HDFS路径加载Excel文件。代码示例中创建了SparkSession，指定了WareHouse目录，并读取了名为`ss.xlsx`的Excel文件，展示其内容。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark读取Excel文件：

pom.xml文件：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cn.wgy.project</groupId>
    <artifactId>ProjectTest0428</artifactId>
    <version>1.0-SNAPSHOT</version>
        <dependencies>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming_2.11</artifactId>
                <version>2.3.0</version>
            </dependency>

            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>5.1.27</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-hive_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-graphx_2.11</artifactId>
                <version>2.3.0</version>
            </dependency>
            <!-- https://mvnrepository.com/artifact/com.cloudera.sparkts/sparkts -->
            <dependency>
                <groupId>com.cloudera.sparkts</groupId>
                <artifactId>sparkts</artifactId>
                <version>0.4.0</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>
            <dependency>
                <groupId>com.alibaba</groupId>
                <artifactId>fastjson</artifactId>
                <version>1.2.75</version>
            </dependency>
            <dependency>
                <groupId>com.crealytics</groupId>
                <artifactId>spark-excel_2.11</artifactId>
                <version>0.13.1</version>
            </dependency>
            <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-collections4 -->
            <dependency>
                <groupId>org.apache.commons</groupId>
                <artifactId>commons-collections4</artifactId>
                <version>4.4</version>
            </dependency>

        </dependencies>
</project>

读取Excel文件：

import org.apache.spark.SparkConf
import org.apache.spark.sql.{DataFrame, DataFrameReader, SparkSession}
import com.crealytics.spark.excel._

object T {
  def main(args: Array[String]): Unit = {
  val frame = getSpark2.read.excel(header = true,inferSchema = true).load("hdfs://hadoop102:9000/ss.xlsx")
    frame.show()
  }


  def getSpark2={
    val sparkSession: SparkSession = SparkSession.builder()
      .config("spark.sql.warehouse.dir","hdfs://hadoop102:9000/user/hive/warehouse")
      .master("local")
      .enableHiveSupport()
      .getOrCreate()
    sparkSession
  }
}