spark-shell到sparkcontext的过程

最新推荐文章于 2023-11-27 23:47:23 发布

原创最新推荐文章于 2023-11-27 23:47:23 发布 · 721 阅读

0 ·

CC 4.0 BY-SA版权

大数据专栏收录该内容

14 篇文章

订阅专栏

本文详细介绍了Apache Spark的启动过程，从spark-shell到spark-class，再到spark-submit，直至创建SparkContext。涉及关键步骤如：调用java运行org.apache.spark.launcher.Main类、设置环境变量等。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >


在bin/spark-shell 中会调用spark-submit
function main() {
  if $cygwin; then
    # Workaround for issue involving JLine and Cygwin
    # (see http://sourceforge.net/p/jline/bugs/40/).
    # If you're using the Mintty terminal emulator in Cygwin, may need to set the
    # "Backspace sends ^H" setting in "Keys" section of the Mintty options
    # (see https://github.com/sbt/sbt/issues/562).
    stty -icanon min 1 -echo > /dev/null 2>&1
    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Djline.terminal=unix"
    "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
    stty icanon echo > /dev/null 2>&1
  else
    export SPARK_SUBMIT_OPTS
    "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"
  fi
}

"bin/spark-submit" 中会继续调用spark-class，并制定执行类org.apache.spark.deploy.SparkSubmit

if [ -z "${SPARK_HOME}" ]; then
  source "$(dirname "$0")"/find-spark-home
fi

# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0

exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
在bin/spark-class中
首先找到runner，这里看到是调用java
# Find the java binary
if [ -n "${JAVA_HOME}" ]; then
  RUNNER="${JAVA_HOME}/bin/java"
else
  if [ "$(command -v java)" ]; then
    RUNNER="java"
  else
    echo "JAVA_HOME is not set" >&2
    exit 1
  fi
fi

可以看到是调用java 运行org.apache.spark.launcher.Main 这个类

build_command() {
  "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
  printf "%d\0" $?
}
下面将build_command 中的保存到CMD
set +o posix
CMD=()
while IFS= read -d '' -r ARG; do
  CMD+=("$ARG")
done < <(build_command "$@")

最后执行java
CMD=("${CMD[@]:0:$LAST}")
exec "${CMD[@]}"


最终在launcher.Main 中调用sparksubmit->repl->sparkloop.process->initializespark->createsparkcontext 至此最重要的
sparkcontext登场了