UDF(UserDefinedFunction)

最新推荐文章于 2025-05-14 16:37:43 发布

原创最新推荐文章于 2025-05-14 16:37:43 发布 · 3.9k 阅读

1 ·

CC 4.0 BY-SA版权

Spark 专栏收录该内容

3 篇文章

订阅专栏

UDF类似于map或lambda，其定义的函数可用于Spark中的DataFrame中行数据的处理。

val df = Seq(
      (1, "boy", "裤子"),
      (2, "girl", "裤子"),
      (3, "boy", "裙子"),
      (4, "girl", "裙子"),
      (5, "girl", "裙子")
    ).toDF("id", "sex", "dressing")
def addpre(pre: String) = udf{ (xing: String, dress: String) => xing + pre + dress  }
df.select($"id", $"sex", $"dressing", addpre(" dresses ")($"sex", $"dressing")).show()
// 运行结果
+---+----+--------+------------------+
| id| sex|dressing|UDF(sex, dressing)|
+---+----+--------+------------------+
|  1| boy|      裤子|    boy dresses 裤子|
|  2|girl|      裤子|   girl dresses 裤子|
|  3| boy|      裙子|    boy dresses 裙子|
|  4|girl|      裙子|   girl dresses 裙子|
|  5|girl|      裙子|   girl dresses 裙子|
+---+----+--------+------------------+