python执行spark任务sparksql、Spark 并发量测试

转载已于 2023-05-26 16:27:31 修改 · 1.6k 阅读

7 ·

CC 4.0 BY-SA版权

原文链接：https://blog.youkuaiyun.com/qq_29726869/article/details/82757381

文章标签：

#spark #big data

于 2021-11-17 15:42:03 首次发布

Spark 同时被 2 个专栏收录

15 篇文章

订阅专栏

Python

6 篇文章

订阅专栏

本文介绍了一个关于Spark集群并发量的测试案例，通过不同程序设置来评估集群在同一时间内能运行的任务数量及其性能表现。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

安装PySpark

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple pyspark

测试

cd /export/server/spark/bin
./pyspark --master local[*]

import sys

from pyspark.sql import SparkSession

sc = SparkSession.builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .enableHiveSupport() \
    .getOrCreate()

# sc = SparkSession.builder.appName("PysparkExample") \
#     .config ("spark.sql.shuffle.partitions", "50") \
#     .config("spark.driver.maxResultSize","5g") \
#     .config ("spark.sql.execution.arrow.enabled", "true") \
#     .getOrCreate()

# date_range_list = ['20210215', '20201130', '20201214']
date_range_list = ['2021-02-15']


def run_sql_process(run_sqls):
    run_sql_list = run_sqls.split(';')
    for run_sql in run_sql_list:
        if run_sql:
            print("running: %s  " % (run_sql.strip()))
            sc.sql(run_sql)


def mutil_process(sqltxt):
    for date in date_range_list:
        run_sql = sqltxt.replace('${START_DT}', date)
        print(run_sql)
        # 创建进程
        run_sql_process(run_sql)


if __name__ == '__main__':
    sql_file = sys.argv[1]
    arg2 = sys.argv[2]
    if sql_file:  # and os.path.exists(sql_path):
        print("if sql_path:", sql_file)
    # if sql_path is not None:
    #     print("sql_path is not None:") 
    # elif True:
    #     print("Still True")
    # else:
    #     print("False")

    # 读取 sql 文件文本内容 
    # sql = open(sql_file, 'r', encoding='utf8')
    # sqltxt = sql.readlines()
    with open(sql_file) as fr:
        sqltxt = fr.read()
    # print("sqltxt:", sqltxt)
    mutil_process(sqltxt)
    print("程序运行成功！！！！！")

/apache/spark/bin/spark-submit \ 
--master yarn \ 
--deploy-mode client \ 
--queue xxx \ 
--conf spark.pyspark.driver.python=/usr/share/anaconda2/bin/python \ 
--conf spark.pyspark.python=/usr/share/anaconda2/bin/python  \ 
--executor-cores 4 \ 
--executor-memory 20g \ 
--driver-memory 1g \
--num-executors 50 \
/apache/releases/spark-2.3.1.1.1.6-bin-ebay/python/pyspark/tests.py  arg1 arg2