_pickle.PicklingError: Could not serialize object: Exception:

最新推荐文章于 2024-06-11 19:48:53 发布

原创最新推荐文章于 2024-06-11 19:48:53 发布 · 533 阅读

CC 4.0 BY-SA版权

文章标签：

1、详细报错：_pickle.PicklingError: Could not serialize object: Exception: It appears that you are attempting to broadcast an RDD or reference an RDD from an action or transformation. RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(lambda x: rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.

2、错误原因：pyspark中的RDD转换操作使用错误

3、错误代码如下：

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local[*]").setAppName("应用名")
sc = SparkContext(conf=conf)
# 简单回顾下针对RDD的转换操作：
RDD = sc.parallelize(
    ['HDFS', 'YARN', 'MapReduce', 'Hive', 'Pig', 'Mahout', 'HBase', 'Sqoop', 'Flume', 'ZooKeeper', 'Ambari'])
print(RDD.distinct(sc.parallelize(['HDFS', 'MapReduce'])).collect())

4、校正后代码如下：

from pyspark import SparkConf, SparkContext

conf = SparkConf().setMaster("local[*]").setAppName("应用名")
sc = SparkContext(conf=conf)
# 简单回顾下针对RDD的转换操作：
# RDD名.distinct()；RDD名.sample(True/False, 比例值, 种子)；RDD名.intersection(RDD名2)；RDD名.subtract(RDD名2)
RDD = sc.parallelize(
    ['HDFS', 'YARN', 'MapReduce', 'Hive', 'Pig', 'Mahout', 'HBase', 'Sqoop', 'Flume', 'ZooKeeper', 'Ambari'])
print(RDD.distinct().collect())

5、总结：pyspark中的distinct转换操作只针对一个RDD，代码模板为 RDD名.distinct() 效果为去除名为‘RDD名’的RDD中的重复值，莫要与subtract转换操作混淆，我便是因为混淆导致出错