Pig内存管理与数据溢出处理
本文探讨了Pig中内存管理机制，重点介绍了如何分配固定内存用于存储袋，以及何时触发数据溢出并转储到磁盘。详细解释了内存使用情况的决定因素及默认设置。
Memory Management
Pig allocates a fix amount of memory to store bags and spills to disk as soon as the memory limit is reached. This is very similar to how Hadoop decides when to spill data accumulated by the combiner.
The amount of memory allocated to bags is determined by pig.cachedbag.memusage; the default is set to 10% of available memory. Note that this memory is shared across all large bags used by the application.
扫一扫
1952
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
