Spark在集群上提交任务的脚本_中山大学集群提交任务脚本-优快云博客

本文链接：https://blog.youkuaiyun.com/LL9504/article/details/103116688

本文详细介绍了一种在Hadoop环境下使用Spark进行集群部署的方法，包括配置环境变量、指定Hadoop配置目录、设置Spark提交脚本路径及参数，以及如何通过spark-submit命令提交Spark应用程序。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark启动脚本:

###hadoop配置文件
export HADOOP_CONF_DIR=/etc/hadoop/conf
#spark-submit路径
sparksubmits="/opt/cloudera/parcels/SPARK2/bin/spark2-submit"
#jar包所在本地目录
jars="/usr/java/checkpoint/SSE_ST2_ANALYSIS_SPARK.jar"

echo "begin running NewsStreamimgClusterDriver model"

#使用spark-submit提交spark程序
su root -c "$sparksubmits --class cn.com.trs.topic.news.streaming.NewsStreamingClusterDriver \
--master yarn \
--driver-cores 1 \
--driver-memory 2g \
--deploy-mode cluster \
--executor-cores 1 \
--num-executors 16 \
--executor-memory 1g \
--name NewsStreamingClusterDriver \
$jars XZ 4 20 \
"
echo "finished!"

l里面用到的spark2-submit的命令脚本

#!/bin/bash
  # Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
  SOURCE="${BASH_SOURCE[0]}"
  BIN_DIR="$( dirname "$SOURCE" )"
  while [ -h "$SOURCE" ]
  do
    SOURCE="$(readlink "$SOURCE")"
    [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
    BIN_DIR="$( cd -P "$( dirname "$SOURCE"  )" && pwd )"
  done
  BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
  CDH_LIB_DIR=$BIN_DIR/../../CDH/lib
  LIB_DIR=$BIN_DIR/../lib
export HADOOP_HOME=$CDH_LIB_DIR/hadoop

# Autodetect JAVA_HOME if not defined
. $CDH_LIB_DIR/bigtop-utils/bigtop-detect-javahome

exec $LIB_DIR/spark2/bin/spark-submit "$@"