(*:update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml

解决SBT Scala版本冲突
本文介绍了解决SBT项目中因不同依赖引入的Scala版本冲突问题的方法。通过统一Scala版本为2.11.8并调整依赖配置,成功解决了构建错误。

SBT异常信息如下:

[info] Loading project definition from D:\IdeaProjects\spark-practice\project
[info] Set current project to spark-practice (in build file:/D:/IdeaProjects/spark-practice/)
> [info] Defining *:shellPrompt
[info] The new value will be used by no settings or tasks.
[info] Reapplying settings...
[info] Set current project to spark-practice (in build file:/D:/IdeaProjects/spark-practice/)
[info] Defining */*:sbtStructureOutputFile
[info] The new value will be used by no settings or tasks.
[info] Reapplying settings...
[info] Set current project to spark-practice (in build file:/D:/IdeaProjects/spark-practice/)
[info] Defining */*:sbtStructureOptions
[info] The new value will be used by no settings or tasks.
[info] Reapplying settings...
[info] Set current project to spark-practice (in build file:/D:/IdeaProjects/spark-practice/)
[info] Applying State transformations org.jetbrains.sbt.CreateTasks from C:/Users/gyg/.IntelliJIdea2016.3/config/plugins/Scala/launcher/sbt-structure-0.13.jar
[info] Reapplying settings...
[info] Set current project to spark-practice (in build file:/D:/IdeaProjects/spark-practice/)
[error] Modules were resolved with conflicting cross-version suffixes in {file:/D:/IdeaProjects/spark-practice/}spark-practice:
[error]    org.scala-lang.modules:scala-xml _2.12, _2.11
[info] Resolving org.scala-sbt#sbt;0.13.15 ...
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Resolving org.scala-sbt#main;0.13.15 ...
[info] Resolving org.scala-sbt#actions;0.13.15 ...
[info] Resolving org.scala-sbt#classpath;0.13.15 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.6 ...
[info] Resolving org.scala-sbt#interface;0.13.15 ...
[info] Resolving org.scala-sbt#io;0.13.15 ...
[info] Resolving org.scala-sbt#control;0.13.15 ...
[info] Resolving org.scala-sbt#launcher-interface;1.0.1 ...
[info] Resolving org.scala-sbt#completion;0.13.15 ...
[info] Resolving org.scala-sbt#collections;0.13.15 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#api;0.13.15 ...
[info] Resolving org.scala-sbt#classfile;0.13.15 ...
[info] Resolving org.scala-sbt#logging;0.13.15 ...
[info] Resolving org.scala-sbt#process;0.13.15 ...
[info] Resolving org.scala-sbt#compiler-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#relation;0.13.15 ...
[info] Resolving org.scala-sbt#compile;0.13.15 ...
[info] Resolving org.scala-sbt#persist;0.13.15 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.10;0.4.2 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#ivy;0.13.15 ...
[info] Resolving org.scala-sbt#cross;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-sbt#serialization_2.10;0.1.2 ...
[info] Resolving org.scala-lang.modules#scala-pickling_2.10;0.10.1 ...
[info] Resolving org.scalamacros#quasiquotes_2.10;2.0.1 ...
[info] Resolving org.json4s#json4s-core_2.10;3.2.10 ...
[info] Resolving org.json4s#json4s-ast_2.10;3.2.10 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving org.spire-math#jawn-parser_2.10;0.6.0 ...
[info] Resolving org.spire-math#json4s-support_2.10;0.6.0 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt#task-system;0.13.15 ...
[info] Resolving org.scala-sbt#tasks;0.13.15 ...
[info] Resolving org.scala-sbt#tracking;0.13.15 ...
[info] Resolving org.scala-sbt#cache;0.13.15 ...
[info] Resolving org.scala-sbt#testing;0.13.15 ...
[info] Resolving org.scala-sbt#test-agent;0.13.15 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.15 ...
[info] Resolving org.scala-sbt#command;0.13.15 ...
[info] Resolving org.scala-sbt#template-resolver;0.1 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving org.scala-sbt#compiler-interface;0.13.15 ...
[info] Resolving org.scala-sbt#api;0.13.15 ...
[info] Resolving org.scala-sbt#compile;0.13.15 ...
[info] Resolving org.scala-sbt#persist;0.13.15 ...
[info] Resolving org.scala-sbt#process;0.13.15 ...
[info] Resolving org.scala-sbt#classpath;0.13.15 ...
[info] Resolving org.scala-sbt#relation;0.13.15 ...
[info] Resolving org.scalamacros#quasiquotes_2.10;2.0.1 ...
[info] Resolving org.scala-sbt#serialization_2.10;0.1.2 ...
[info] Resolving org.scala-lang.modules#scala-pickling_2.10;0.10.1 ...
[info] Resolving org.scala-sbt#test-agent;0.13.15 ...
[info] Resolving org.scala-sbt#logging;0.13.15 ...
[info] Resolving org.scala-sbt#command;0.13.15 ...
[info] Resolving org.scala-sbt#task-system;0.13.15 ...
[info] Resolving org.scala-sbt#compiler-integration;0.13.15 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.6 ...
[info] Resolving org.scala-sbt#actions;0.13.15 ...
[info] Resolving org.scala-sbt#classfile;0.13.15 ...
[info] Resolving org.scala-sbt#ivy;0.13.15 ...
[info] Resolving org.scala-sbt#compiler-interface;0.13.15 ...
[info] Resolving org.scala-sbt#cache;0.13.15 ...
[info] Resolving org.scala-sbt#testing;0.13.15 ...
[info] Resolving org.scala-sbt#launcher-interface;1.0.1 ...
[info] Resolving org.json4s#json4s-core_2.10;3.2.10 ...
[info] Resolving org.scala-sbt#cross;0.13.15 ...
[info] Resolving org.scala-sbt#tasks;0.13.15 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.15 ...
[info] Resolving org.scala-sbt#main;0.13.15 ...
[info] Resolving org.scala-sbt#collections;0.13.15 ...
[info] Resolving org.spire-math#jawn-parser_2.10;0.6.0 ...
[info] Resolving org.scala-sbt#io;0.13.15 ...
[info] Resolving org.scala-sbt#interface;0.13.15 ...
[info] Resolving org.json4s#json4s-ast_2.10;3.2.10 ...
[info] Resolving org.scala-lang#scala-library;2.10.6 ...
[info] Resolving org.scala-sbt#template-resolver;0.1 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.10;0.4.2 ...
[info] Resolving org.spire-math#json4s-support_2.10;0.6.0 ...
[info] Resolving org.scala-sbt#tracking;0.13.15 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving org.scala-sbt#completion;0.13.15 ...
[info] Resolving org.scala-sbt#control;0.13.15 ...
[info] Resolving org.scala-sbt#sbt;0.13.15 ...
[info] Resolving org.scala-sbt#run;0.13.15 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-48dd0744422128446aee9ac31aa356ee203cc9f4 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.6 ...
[info] Resolving jline#jline;2.14.3 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.15 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.15 ...
[info] Resolving org.scala-sbt#logic;0.13.15 ...
[info] Resolving org.scala-sbt#main-settings;0.13.15 ...
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[error] (*:update) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml
[error] (*:ssExtractDependencies) Conflicting cross-version suffixes in: org.scala-lang.modules:scala-xml
[error] Total time: 1 s, completed 2017-5-18 1:02:48
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384M; support was removed in 8.0

我的sbt配置文件

```
name := "spark"

version := "1.0"

scalaVersion := "2.12.2"

resolvers += "Cloudera Repository" at "https://repository.cloudera.com/content/repositories/releases/"
resolvers += "Spring Plugins Repository" at "http://repo.spring.io/plugins-release/"
resolvers += "Maven Repository" at "https://mvnrepository.com/"

libraryDependencies += "org.apache.camel" % "camel-test-spring" % "2.10.1" % "test" intransitive()

libraryDependencies += "com.google.guava" % "guava" % "16.0.1"
libraryDependencies += "commons-codec" % "commons-codec" % "1.10"
libraryDependencies += "commons-logging" % "commons-logging" % "1.2"
libraryDependencies += "org.codehaus.jackson" % "jackson-mapper-asl" % "1.9.13"
libraryDependencies += "jline" % "jline" % "2.12.1"

libraryDependencies += "org.apache.zookeeper" % "zookeeper" % "3.4.5-cdh5.7.1"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-auth" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-annotations" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-server-nodemanager" % "2.6.0-cdh5.7.1"

libraryDependencies += "org.apache.hbase" % "hbase" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.7.1"


libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.12.2"
libraryDependencies += "com.typesafe.scala-logging" % "scala-logging_2.11" % "3.5.0" % "provided"

libraryDependencies += "org.mongodb" % "casbah_2.11" % "3.1.1"
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"

出现error的地方字面理解是Scala版本冲突,即用了2.11版本的Scala,也用了2.12版本的Scala。但是到写这篇博客为止,去mvn中没有找到2.12版本的spark,所以放弃2.12.x版本的Scala,使用了2.11.8之后刷新sbt工程后,下载jar包正常也没有在报错。。

例如上面的
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
其依赖的是
spark-core_2.11

总结一下:
所有jar包依赖的Scala版本号必须一致,不仅spark,还有mongodb等等。
以2.11结尾的jar(如spark-core_2.11)都依赖2.11.x版本的Scala,所以必须用2.11.x版本的Scala,否则会出现Scala包依赖冲突

修改后好用的sbt文件如下:

name := "spark"

version := "1.0"

scalaVersion := "2.11.8"

resolvers += "Cloudera Repository" at "https://repository.cloudera.com/content/repositories/releases/"
resolvers += "Spring Plugins Repository" at "http://repo.spring.io/plugins-release/"
resolvers += "Maven Repository" at "https://mvnrepository.com/"

//  所有jar包依赖的Scala版本号必须一致,
//  即所有以2.11结尾的jar(如spark-core_2.11)都依赖2.11.x版本的Scala,所以必须用2.11.x版本的Scala,否则会出现Scala包依赖冲突

libraryDependencies += "org.apache.camel" % "camel-test-spring" % "2.10.1" % "test" intransitive()

libraryDependencies += "com.google.guava" % "guava" % "16.0.1"
libraryDependencies += "commons-codec" % "commons-codec" % "1.10"
libraryDependencies += "commons-logging" % "commons-logging" % "1.2"
libraryDependencies += "org.codehaus.jackson" % "jackson-mapper-asl" % "1.9.13"
libraryDependencies += "jline" % "jline" % "2.12.1"

libraryDependencies += "org.apache.zookeeper" % "zookeeper" % "3.4.5-cdh5.7.1"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-auth" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-annotations" % "2.6.0-cdh5.7.1"
libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-server-nodemanager" % "2.6.0-cdh5.7.1"

libraryDependencies += "org.apache.hbase" % "hbase" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.7.1"
libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.7.1"


libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.8"
libraryDependencies += "com.typesafe.scala-logging" % "scala-logging_2.11" % "3.5.0" % "provided"

libraryDependencies += "org.mongodb" % "casbah_2.11" % "3.1.1"
libraryDependencies += "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
在使用 `argparse` 模块添加命令行参数时,如果出现类似 `argparse.ArgumentError: argument --preprocess: conflicting option string: --preprocess` 的错误,说明当前尝试添加的参数名称已经存在于解析器中。这通常由以下几种原因导致: 1. **重复添加相同参数**:同一个参数名被多次调用 `add_argument()` 添加到解析器中。 2. **与其他模块或库冲突**:某些第三方库可能也注册了相同的参数名,尤其是在大型项目或框架中。 3. **继承或复用已有解析器对象**:在复用 `ArgumentParser` 或 `Namespace` 对象时,未清理旧的参数定义。 ### 解决方法 - **检查代码中是否重复添加参数** 确保在代码中没有多次为 `--preprocess` 添加定义。例如: ```python parser.add_argument('--preprocess', type=str, help='Path to preprocessing script') ``` 如果这段代码被执行多次,就会引发冲突。可以通过在添加前检查是否已存在该参数: ```python if '--preprocess' not in sys.argv: parser.add_argument('--preprocess', type=str, help='Path to preprocessing script') ``` - **使用 `conflict_handler='resolve'`** 在创建 `ArgumentParser` 实例时,可以设置冲突处理方式为自动覆盖之前的定义[^1]: ```python parser = argparse.ArgumentParser(conflict_handler='resolve') parser.add_argument('--preprocess', type=str, help='First definition') parser.add_argument('--preprocess', type=str, help='Second definition (will override)') ``` 这样后续的定义将覆盖前面的定义,适用于需要动态修改参数定义的场景。 - **避免与第三方库冲突** 如果问题出现在使用某个库(如 `fairseq`)之后,则可能是该库内部已经注册了同名参数。可以在导入相关模块之前就完成自定义参数的添加,或者通过命名空间隔离的方式管理参数[^1]。 - **使用子解析器或独立的 ArgumentParser 实例** 将不同功能模块的参数分离到不同的解析器中,避免全局污染: ```python main_parser = argparse.ArgumentParser() preprocess_parser = main_parser.add_argument_group('Preprocessing Options') preprocess_parser.add_argument('--preprocess', type=str) ``` ### 示例代码 以下是一个完整的示例,展示如何安全地添加参数并避免冲突: ```python import argparse import sys def add_preprocess_arg(parser): if '--preprocess' not in sys.argv: parser.add_argument('--preprocess', type=str, help='Path to preprocessing script') def main(): parser = argparse.ArgumentParser(conflict_handler='resolve') add_preprocess_arg(parser) args = parser.parse_args() print(args) if __name__ == '__main__': main() ``` 通过上述方法,可以有效解决 `argparse` 中因参数名重复而导致的冲突问题。 ---
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值