PySpark完美安装

一、hadoop版本号确认

1. hadoop ==> 2.7.2

[root@dm46 TDH-Client]# hadoop version

Hadoop 2.7.2-transwarp-6.2.0

Subversion http://xxxx:10080/hadoop/hadoop-2.7.2-transwarp.git -r f31230971c2a36e77e4886e0f621366826cec3a3

Compiled by jenkins on 2019-07-27T11:33Z

Compiled with protoc 2.5.0

二、下载spark

注意：选择 Pre-built for Apache Hadoop 2.7

https://dlcdn.apache.org/spark/spark-3.2.3/spark-3.2.3-bin-hadoop2.7.tgz

三、准备pyspark环境

1. 创建虚拟环境

conda create -n pyspark_env python=3.8

conda activate pyspark_env

2. 安装pyspark

注意：确保spark版本与pyspark版本一致

pip install pyspark==3.2.3 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

3.测试

cd /home/xxx/kdh/spark/bin

./spark-submit --master local[*] /home/xxx/kdh/spark/examples/src/main/python/pi.py 30

四、常见报错

1. RuntimeError: Python in worker has different version 3.8 than that in driver 3.6

原因：local模式下，此时使用的是当前环境的python。如果代码里指定了pyspark_env，当前环境也应该是pyspark_env。

conda activate pyspark_env

cd /home/xxx/kdh/spark/bin

./spark-submit --master local[*] /home/xxx/workspace/pyspark_learn/02_pyspark_core/main/02_Wordcount_hdfs.py

五、其它

1. arrow安装

pip install pyspark[sql] -i https://pypi.tuna.tsinghua.edu.cn/simple