运行Spark项目出错error: error while loading <root>, error in opening zip file

最新推荐文章于 2024-04-05 00:15:00 发布

原创最新推荐文章于 2024-04-05 00:15:00 发布 · 1.8k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #Invalid POM for *.jar

Spark 专栏收录该内容

1 篇文章

订阅专栏

本文记录了一个初学者在使用Maven构建Spark WordCount项目时遇到的问题及解决方案，主要问题是由于依赖库commons-lang和commons-codec的POM文件无效导致构建失败。

刚学spark，想写一个wordcount的，结果一上来就是一个坑！！！！

具体pom.xml文件如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.jxy.test</groupId>
    <artifactId>day28</artifactId>
    <version>1.0</version>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.10.6</scala.version>
        <spark.version>1.6.1</spark.version>
        <hadoop.version>2.6.4</hadoop.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-make:transitive</arg>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>2.10.6</scalaVersion>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.jxy.test.WordCount</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

结果mvn install的时候，报如下错误：

[WARNING] Invalid POM for commons-codec:commons-codec:jar:1.3, transitive dependencies (if any) will not be available, enable
 debug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[INFO] F:\Hadoop_shizhan\Project\day28\src\main\scala:-1: info: compiling
[INFO] Compiling 1 source files to F:\Hadoop_shizhan\Project\day28\target\classes at 1525448868996
[ERROR] error: error while loading <root>, error in opening zip file
[ERROR] error: scala.reflect.internal.MissingRequirementError: object scala.runtime in compiler mirror not found.
[ERROR]         at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[ERROR]         at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getPackage(Mirrors.scala:172)
[ERROR]         at scala.reflect.internal.Mirrors$RootsBase.getRequiredPackage(Mirrors.scala:175)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage$lzycompute(Definitions.scala:183)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackage(Definitions.scala:183)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass$lzycompute(Definitions.scala:184)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.RuntimePackageClass(Definitions.scala:184)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr$lzycompute(Definitions.scala:102
4)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.AnnotationDefaultAttr(Definitions.scala:1023)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses$lzycompute(Definitions.scala:1153
)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.syntheticCoreClasses(Definitions.scala:1152)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode$lzycompute(Definitions.sca
la:1196)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.symbolsNotPresentInBytecode(Definitions.scala:1196)
[ERROR]         at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1261)
[ERROR]         at scala.tools.nsc.Global$Run.<init>(Global.scala:1290)
[ERROR]         at scala.tools.nsc.Driver.doCompile(Driver.scala:32)
[ERROR]         at scala.tools.nsc.Main$.doCompile(Main.scala:79)
[ERROR]         at scala.tools.nsc.Driver.process(Driver.scala:54)
[ERROR]         at scala.tools.nsc.Driver.main(Driver.scala:67)
[ERROR]         at scala.tools.nsc.Main.main(Main.scala)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR]         at scala_maven_executions.MainHelper.runMain(MainHelper.java:164)
[ERROR]         at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)
[ERROR]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

困惑了我一晚上，开始参考了如下几个博客：

https://blog.youkuaiyun.com/bitcarmanlee/article/details/79170716

https://blog.youkuaiyun.com/u010638969/article/details/62224708

都说是maven库有问题，于是将最新下载的maven库文件夹都删除了，基本上是跟scala有关的，发现还是不行

快要放弃的时候，重新看了一下控制台日志，发现有如下几行日志：

[WARNING] Invalid POM for commons-codec:commons-codec:jar:1.3, transitive dependencies (if any) will not be available, enable
 debug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[WARNING] Invalid POM for commons-lang:commons-lang:jar:2.6, transitive dependencies (if any) will not be available, enable d
ebug logging for more details
[INFO] F:\Hadoop_shizhan\Project\day28\src\main\scala:-1: info: compiling

因为一开始看是warning，就没注意！结果这才是问题出现的关键。它的意思大致就是说你的commons-lang.jar、commons-codec.jar无效！

进入maven仓库，将这两个jar删了，重新mvn install一下，成功，问题解决