JAVA代码实现编程式提交Spark任务

最新推荐文章于 2024-05-21 14:54:56 发布

浅汐王

最新推荐文章于 2024-05-21 14:54:56 发布

阅读量7w

点赞数

分类专栏： spark

本文链接：https://blog.youkuaiyun.com/qq_32252917/article/details/107666785

版权

spark 专栏收录该内容

34 篇文章

订阅专栏

本文记录了三种使用Java编程式提交Spark任务的方法：1) 直接调用SparkSubmit的main方法；2) 使用SparkLauncher的launch或startApplication方法；3) 通过RestSubmissionClient的run方法。特别指出，使用RestSubmissionClient时注意Rest URL端口为6066。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

三种方法作为记录：

1）直接调用SparkSubmit的main方法

2）SparkLauncher类的launch方法或者startApplication方法

3）使用RestSubmissionClient的run方法

SparkSubmit提交任务

	String[] param = {
				"--class", "org.apache.spark.examples.SparkPi",
				"--master", "local[2]",
				"/bigdata/spark-2.4.6-bin-hadoop2.6/examples/jars/spark-examples_2.11-2.4.6.jar",
		};

		SparkSubmit.main(param);

		System.out.println("任务运行完成。。。。。。。。。");

可见，基本上就是把spark-submit脚本放到了java中执行！

但是要注意：（Rest URL端口是6066）

 "--master", "spark://192.168.0.181:6066"

SparkLauncher提交任务

public static void main(String[] args) throws Exception {
        HashMap<String, String> envParams = new HashMap<>();
        envParams.put("YARN_CONF_DIR", "/home/hadoop/cluster/hadoop-release/etc/hadoop");
        envParams.put("HADOOP_CONF_DIR", "/home/hadoop/cluster/hadoop-release/etc/hadoop");
        envParams.put("SPARK_HOME", "/home/hadoop/cluster/spark-new");
        envParams.put("SPARK_PRINT_LAUNCH_COMMAND", "1");

        SparkAppHandle spark = new SparkLauncher(envParams)
                .setAppResource("/home/hadoop/cluster/spark-new/examples/jars/spark-examples_2.11-2.2.1.jar")
                .setMainClass("org.apache.spark.examples.SparkPi")
                .setMaster("yarn")
                .startApplication();

        Thread.sleep(100000);
    }

RestSubmissionClient的run方法提交

@Test
    public void submit() {
        String appResource = "hdfs://192.168.0.181:8020/opt/guoxiang/wordcount.jar";
        String mainClass = "com.fly.spark.WordCount";
        String[] args = {
                "hdfs://192.168.0.181:8020/opt/guoxiang/wordcount.txt",
                "hdfs://192.168.0.181:8020/opt/guoxiang/wordcount"
        };

        SparkConf sparkConf = new SparkConf();
        // 下面的是参考任务实时提交的Debug信息编写的
        sparkConf.setMaster("spark://192.168.0.181:6066")
                .setAppName("carabon" + " " + System.currentTimeMillis())
                .set("spark.executor.cores", "4")
                .set("spark.submit.deployMode", "cluster")
                .set("spark.jars", appResource)
                .set("spark.executor.memory", "1g")
                .set("spark.cores.max", "4")
                .set("spark.driver.supervise", "false");
        Map<String, String> env = System.getenv();

        CreateSubmissionResponse response = null;
        try {
            response = (CreateSubmissionResponse)
                    RestSubmissionClient.run(appResource, mainClass, args, sparkConf, new HashMap<>());
        } catch (Exception e) {
            e.printStackTrace();
        }
        System.out.println(response.toJson());
    }