pypspark udf函数传入参数

我是女孩

于 2020-09-10 19:34:41 发布

阅读量863

点赞数

分类专栏：大数据spark

本文链接：https://blog.youkuaiyun.com/u013385018/article/details/108521314

版权

大数据spark 专栏收录该内容

11 篇文章

订阅专栏

本文介绍了一种在 Apache Spark 中使用 User Defined Function (UDF) 的方法，具体展示了如何将数据帧列和外部列表传递给 UDF，并在 PySpark 环境下实现对特定主题关键词计数的功能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

https://stackoverflow.com/questions/37409857/passing-a-data-frame-column-and-external-list-to-udf-under-withcolumn

keyword_list=[
    ['union','workers','strike','pay','rally','free','immigration',],
    ['farmer','plants','fruits','workers'],
    ['outside','field','party','clothes','fashions']]

def label_maker_topic(tokens, topic_words_broadcasted):
    twt_list = []
    for i in range(0, len(topic_words_broadcasted.value)):
        count = 0
        #print(topic_words[i])
        for tkn in tokens:
            if tkn in topic_words_broadcasted.value[i]:
                count += 1
        twt_list.append(count)

    return twt_list

def make_topic_word_better(topic_words_broadcasted):
    def f(c):
        return label_maker_topic(c, topic_words_broadcasted)
    return F.udf(f)

df = spark.createDataFrame([["union",], ["party",]]).toDF("tokens")
b = spark.sparkContext.broadcast(keyword_list)
df.withColumn("topics", make_topic_word_better(b)(F.col("tokens"))).show()