Spark Submit提交时，Json字符串作为参数

最新推荐文章于 2025-01-17 18:36:59 发布

南修子

最新推荐文章于 2025-01-17 18:36:59 发布

阅读量904

点赞数 2

CC 4.0 BY-SA版权

分类专栏： spark 文章标签： spark json

本文链接：https://blog.youkuaiyun.com/huang_rx/article/details/106788386

spark 专栏收录该内容

7 篇文章

订阅专栏

本文讨论了在Spark Submit过程中遇到的JSON字符串被替换导致的问题，源码分析揭示了}}和{{被移除的原因，使得传入的JSON变为无效。文章介绍了作者所采用的解决方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

#今天遇到一个把json作为参数传入spark程序中的问题
###原因如下，Spark 源码中会对把}} 和 {{替换掉

@VisibleForTesting
  public static String expandEnvironment(String var,
      Path containerLogDir) {
    var = var.replace(ApplicationConstants.LOG_DIR_EXPANSION_VAR,
      containerLogDir.toString());
    var =  var.replace(ApplicationConstants.CLASS_PATH_SEPARATOR,
      File.pathSeparator);

    // replace parameter expansion marker. e.g. {{VAR}} on Windows is replaced
    // as %VAR% and on Linux replaced as "$VAR"
    if (Shell.WINDOWS) {
      var = var.replaceAll("(\\{\\{)|(\\}\\})", "%");
    } else {
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_LEFT, "$");
      var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");
    }
    return var;
  }

###造成的结果就是，json传到程序中，已经变成一个不完整的JSON字符串了

20/06/04 12:21:14 ERROR yarn.ApplicationMaster: User class threw exception: com.alibaba.fastjson.JSONException: syntax error
com.alibaba.fastjson.JSONException: syntax error
	at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:528)
	at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:564)
	at com.alibaba.fastjson.parser.DefaultJSONParser.parse(DefaultJSONParser.java:1400)
	at com.alibaba.fastjson.parser.DefaultJSONParser.parse(DefaultJSONParser.java:1366)
	at com.alibaba.fastjson.JSON.parse(JSON.java:170)
	at com.alibaba.fastjson.JSON.parse(JSON.java:180)
	at com.alibaba.fastjson.JSON.parse(JSON.java:149)
	at com.alibaba.fastjson.JSON.parseObject(JSON.java:241)
	at cn.richinfo.dataclear.operator.SingleTableDataCleaningNew.enter(SingleTableDataCleaningNew.java:61)
	at cn.richinfo.dataclear.operator.MainEntrance.main(MainEntrance.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:646)
20/06/04 12:21:14 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: com.alibaba.fastjson.JSONException: syntax error)

###我的解决方案就是

在}}中间加上一个空格} },之后，就不会被替换掉

Especially this line var = var.replace(ApplicationConstants.PARAMETER_EXPANSION_RIGHT, "");. The right braces will be replaced with empty String.

So this is actually not a Spark issue, and may not easily be fixed by Yarn. But you can have many workarounds to avoid passing json String with arguments.

网上这个人说的，还有别的方式比较好的，就不得而知了