Spark 启动 java.lang.ClassNotFoundException: parquet.hadoop.ParquetOutputCommitter

最新推荐文章于 2023-09-14 20:59:17 发布

原创最新推荐文章于 2023-09-14 20:59:17 发布 · 4.1k 阅读

CC 4.0 BY-SA版权

本文为博主原创文章，未经博主允许不得转载。

本文介绍了解决Spark启动时报java.lang.ClassNotFoundException: parquet.hadoop.ParquetOutputCommitter错误的方法。通过下载特定版本的parquet-hadoop jar包并将其添加到Spark的ClassPath中，成功解决了该问题。

Spark启动报 java.lang.ClassNotFoundException: parquet.hadoop.ParquetOutputCommitter
我安装的是hadoop-2.6.0-cdh5.12.1和spark-1.6.0-cdh5.12.1
解决的版本是
将下面的jar包下载下来放到Spark的启动ClassPath下,然后重启Spark

<dependency>
    <groupId>com.twitter</groupId>
    <artifactId>parquet-hadoop</artifactId>
    <version>1.4.3</version>
</dependency>

执行

bin/spark-shell

可以正常启动

levin@Levin-PC:~/install-soft/bigdata/spark-1.6.0-cdh5.12.1/bin$ ./spark-shell 
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.
17/09/19 16:40:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/09/19 16:40:29 WARN util.Utils: Your hostname, Levin-PC resolves to a loopback address: 127.0.1.1; using 10.236.103.148 instead (on interface enp0s3)
17/09/19 16:40:29 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Spark context available as sc (master = local[*], app id = local-1505810430958).
SQL context available as sqlContext.
scala>