spark-defaults.conf 里的参数被加载的时机

最新推荐文章于 2025-05-13 15:23:12 发布

原创最新推荐文章于 2025-05-13 15:23:12 发布 · 4k 阅读

0 ·

CC 4.0 BY-SA版权

spark 专栏收录该内容

158 篇文章

订阅专栏

本文探讨了Spark中spark-defaults.conf配置文件的加载时机与过程。通过深入分析源代码，揭示了该配置文件如何被Spark应用程序读取并应用于实际运行环境中。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

我在调试Spark参数时，发现里的参数是默认值，把saprk/conf目录加到source也没有用。我查了一年，spark-shell命令执行时只加载spark-env.sh文件，那么spark-defaults.conf里的内容是什么时候加载的。

在org.apache.spark.launcher.CommandBuilderUtils.java 里有spark-defaults.conf的一个常量DEFAULT_PROPERTIES_FILE等于default-defaults.conf

 static final String DEFAULT_PROPERTIES_FILE = "spark-defaults.conf";

在org.apache.spark.launcher.AbstractCommandBuilder.java 里的loadPropertiesFile()里读取DEFAULT_PROPERTIES_FILE文件

/**
   * Loads the configuration file for the application, if it exists. This is either the
   * user-specified properties file, or the spark-defaults.conf file under the Spark configuration
   * directory.
   */
  private Properties loadPropertiesFile() throws IOException {
    Properties props = new Properties();
    File propsFile;
    if (propertiesFile != null) {
      propsFile = new File(propertiesFile);
      checkArgument(propsFile.isFile(), "Invalid properties file '%s'.", propertiesFile);
    } else {
      propsFile = new File(getConfDir(), DEFAULT_PROPERTIES_FILE);
    }

    if (propsFile.isFile()) {
      FileInputStream fd = null;
      try {
        fd = new FileInputStream(propsFile);
        props.load(new InputStreamReader(fd, "UTF-8"));
        for (Map.Entry<Object, Object> e : props.entrySet()) {
          e.setValue(e.getValue().toString().trim());
        }
      } finally {
        if (fd != null) {
          try {
            fd.close();
          } catch (IOException e) {
            // Ignore.
          }
        }
      }
    }

    return props;
  }

然后laodPropertiesFile()方法被getEffectiveConfig方法使用。

Map<String, String> getEffectiveConfig() throws IOException {
    if (effectiveConfig == null) {
      effectiveConfig = new HashMap<>(conf);
      Properties p = loadPropertiesFile();
      for (String key : p.stringPropertyNames()) {
        if (!effectiveConfig.containsKey(key)) {
          effectiveConfig.put(key, p.getProperty(key));
        }
      }
    }
    return effectiveConfig;
  }

AbstractCommandBuilder 被以下地方使用。

./launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:class SparkSubmitCommandBuilder extends AbstractCommandBuilder {
./launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java:class SparkClassCommandBuilder extends AbstractCommandBuilder {
./launcher/src/main/java/org/apache/spark/launcher/Main.java: AbstractCommandBuilder builder;