Pycharm 搭建pyspark开发环境

Shyllin

已于 2023-12-29 10:19:55 修改

阅读量4k

点赞数 3

分类专栏： Spark 文章标签： pycharm ide 大数据 spark python

于 2019-06-03 20:40:50 首次发布

本文链接：https://blog.youkuaiyun.com/Shyllin/article/details/90758646

版权

Spark 专栏收录该内容

13 篇文章

订阅专栏

本文详细介绍了如何使用PyCharm搭建PySpark开发环境，包括Spark的下载、安装及验证，以及在PyCharm中配置虚拟环境和项目变量的过程。通过示例代码展示了如何执行WordCount任务，并在本地运行Spark应用程序。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Pycharm 搭建pyspark开发环境

spark安装

spark下载

下载地址 http://spark.apache.org/downloads.html

本次是搭建环境使用的官网已编译的版本，如需自己编译可参照官网自行编译，地址为 http://spark.apache.org/docs/latest/building-spark.html

验证spark是否安装成功

(spark_demo)  shylin  ~/Desktop/work/spark_demo  cd ~/Downloads/spark-2.4.0-bin-hadoop2.7/bin
(spark_demo)  shylin  ~/Downloads/spark-2.4.0-bin-hadoop2.7/bin  ./pyspark 
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018, 19:50:54) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
2019-05-24 10:55:19 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Python version 3.6.6 (v3.6.6:4cf1f54eb7, Jun 26 2018 19:50:54)
SparkSession available as 'spark'.
>>>

搭建pyspark开发环境

打开pycharm，新建project，创建一个新的虚拟环境
配置项目环境变量，方便每次创建新py文件都要再次环境变量，操作如下图所示

在这里插入图片描述

SPARK_HOME /Users/shylin/Downloads/spark-2.4.0-bin-hadoop2.7(spark的下载目录)

添加两个包到项目目录下包的路径如下： /Users/shylin/Downloads/spark-2.4.0-bin-hadoop2.7/python/lib
执行wordcount

# word_count.py

#!/usr/bin/python
# -*- coding: UTF-8 -*-

from pyspark.sql import SparkSession

if __name__ == '__main__':

    spark = SparkSession.builder.appName("test").getOrCreate()
    sc = spark.sparkContext
		# file:// 指明是本地文件，默认是找hdfs文件
    counts = sc.textFile('file:///Users/shylin/Desktop/work/spark_demo/test.txt') \
            .flatMap(lambda line: line.split(" ")) \
            .map(lambda x: (x, 1)) \
            .reduceByKey(lambda a, b: a + b)

    output = counts.collect()

    for (word, count) in output:
        print("%s: %i" % (word, count))

    sc.stop()

执行结果

/Users/shylin/.virtualenvs/spark_demo/bin/python /Users/shylin/Desktop/work/spark_demo/word_count.py
2019-05-24 11:23:05 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
[Stage 0:>                                                          (0 + 2) / 2]/Users/shylin/Downloads/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:60: UserWarning: Please install psutil to have better support with spilling
/Users/shylin/Downloads/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/shuffle.py:60: UserWarning: Please install psutil to have better support with spilling
world: 2
python: 3
hadoop: 1
hello: 4
spark: 1

Process finished with exit code 0

以上结果显示一个警告， pip install psutil 即可消除警告

服务器上提交spark 任务

sudo -uhive spark-submit --master local[4] word_count.py

Shylin