shell spark-submit提交之后获取appid，并在程序中扫描状态

使用shell获取spark-submit的appid并监控状态

最新推荐文章于 2024-07-02 14:37:51 发布

原创最新推荐文章于 2024-07-02 14:37:51 发布 · 2.2k 阅读

4 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据

spark 专栏收录该内容

7 篇文章

订阅专栏

本文介绍了如何通过shell脚本提交spark作业，并在提交后获取application_id。接着，利用YARN的REST API来监控作业状态，以application_1591596770810_0207为例展示了具体操作。

#首先看一下提交脚本

#/sbin/bash
/opt/cloudera/parcels/SPARK2/bin/spark2-submit \ 
export JAVA_HOME=/opt/jdk1.8.0_31
TASK_TYPE=$1
TASK_JSON=$2
SPARK_CONFIG=$3
appId=`
$SPARK_CONFIG \
"$TASK_JSON" \
$TASK_TYPE \
2>&1 | tee /dev/tty | grep -i "Submitting application" | grep -o  application_[0-#9]*_[0-9]*`
echo "appid:"$appId

就是下面这两个我们输出格式application_1591596770810_0207
appId=2>&1 | tee /dev/tty | grep -i "Submitting application" | grep -o application_[0-#9]*_[0-9]*
echo “appid:”$appId

###利用yarn提供的restapi去取

//yarnaddress: http://*****:8088/ws/v1/cluster/apps
    private String readAll(Reader rd) throws IOException {

        StringBuilder sb = new StringBuilder();

        int cp;

        while ((cp = rd.read()) != -1) {

            sb.append((char) cp);

        }

        return sb.toString();

    }


    public JSONObject readJsonFromUrl(String url) throws IOException, JSONException {

        InputStream is = new URL(url).openStream();

        try {

            BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));

            String jsonText = readAll(rd);

            JSONObject json = JSONObject.parseObject(jsonText);// new JSONObject(jsonText);

            return json;

        } finally {

            is.close();

        }

    }

###以下值提供参考

 JSONObject info = app.getJSONObject(i);

            String id = info.getString("id");
           // if (!id.equals("application_1591164470599_0150")) continue;

            System.out.println("appid:" + id);

            String user = info.getString("user");

            System.out.println("user:" + user);

            String name = info.getString("name");

            System.out.println("job name:" + name);

            String queue = info.getString("queue");

            System.out.println("job queue:" + queue);

            String state = info.getString("state");

            System.out.println("job state:" + state);

            String finalStatus = info.getString("finalStatus");

            System.out.println("job finalStatus:" + finalStatus);

            int progress = info.getInteger("progress");

            System.out.println("job progress:" + progress);

            String applicationType = info.getString("applicationType");

            System.out.println("job applicationType:" + applicationType);

            long startedTime = info.getLong("startedTime");

            System.out.println("job startedTime:" + startedTime);

            long finishedTime = info.getLong("finishedTime");

            System.out.println("job finishedTime:" + finishedTime);

            long elapsedTime = info.getLong("elapsedTime");

            System.out.println("job elapsedTime:" + elapsedTime);

            System.out.println("-------------------------------------------------------------------------------------------------------");

#####文出自，南修子，转载请注明