Spark-SQL-Python编程

最新推荐文章于 2024-03-07 14:44:34 发布

原创

最新推荐文章于 2024-03-07 14:44:34 发布 · 1.1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#spark-sql #python #sql #spark #pycharm

这篇博客详细介绍了如何在PyCharm环境下利用Python进行Spark SQL编程，涵盖了设置环境、创建DataFrame、执行SQL查询等关键步骤。

使用Pycharm来实现Spark-SQL。

from pyspark import Row
from pyspark.sql import SparkSession
from pyspark.sql.types import StructField, StringType, StructType

if __name__ == "__main__":
    spark = SparkSession\
            .builder\
            .appName("app name")\
            .master("local")\
            .getOrCreate()
    sc = spark.sparkContext
    line = sc.textFile("D:\\data\\demo.txt").map(lambda x: x.split('|'))
    # personRdd = line.map(lambda p: Row(id=p[0], name=p[1], age=int(p[2])))
    # personRdd_tmp = spark.createDataFrame(personRdd)
    # personRdd_tmp.show()

    #读取数据
    schemaString = "id name age"
    fields = list(map(lambda fieldName: StructField(fieldName, StringType(), nullable=True), schemaString.split(" ")))
    schema = StructType(fields)

    rowRDD = line.map(lambda attributes: Row(attributes[0], attributes[1],attributes[2]))
    peopleDF = spark.createDataFrame(rowRDD, schema)
    peopleDF.createOrReplaceT