What does Closure.cleaner (func) mean in Spark?

最新推荐文章于 2025-11-23 10:00:30 发布

转载最新推荐文章于 2025-11-23 10:00:30 发布 · 145 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://www.quora.com/unanswered/Apache-Spark/What-does-Closure-cleaner-func-mean-in-Spark

文章标签：

#big data #spark

大数据专栏收录该内容

16 篇文章

订阅专栏

为解决Scala构造闭包时捕获过多外部变量的问题，Spark引入了闭包清理机制。该机制可在运行时准确地移除不必要的引用，从而避免网络带宽浪费并确保闭包能被正确序列化。

Ankur Dave, Spark committer at UCB AMPLab
Answered Nov 26, 2013
When Scala constructs a closure, it determines which outer variables the closure will use and stores references to them in the closure object. This allows the closure to work properly even when it’s called from a different scope than it was created in.

Scala sometimes errs on the side of capturing too many outer variables (see SI-1419). That’s harmless in most cases, because the extra captured variables simply don’t get used (though this prevents them from getting GC’d). But it poses a problem for Spark, which has to send closures across the network so they can be run on slaves. When a closure contains unnecessary references, it wastes network bandwidth. More importantly, some of the references may point to non-serializable objects, and Spark will fail to serialize the closure.

To work around this bug in Scala, the ClosureCleaner traverses the object at runtime and prunes the unnecessary references. Since it does this at runtime, it can be more accurate than the Scala compiler can. Spark can then safely serialize the cleaned closure.