How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "

chenfangfang_2015

于 2015-09-24 16:53:14 发布

阅读量4.9k

点赞数 1

CC 4.0 BY-SA版权

文章标签： spark StoreMemory spark streaming

本文链接：https://blog.youkuaiyun.com/lulynn/article/details/48713415

本文介绍了如何解决Spark Streaming在处理数据时遇到的'Not enough space to cache input-0-* in memory!'错误。通过增加`spark.storage.memoryFraction`的值到0.9以及将`spark.driver.memory`从1g提升到2g，成功解决了此内存不足的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Error:
15/09/22 21:52:51 WARN storage.MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block input-0-1442929965600 in memory.
15/09/22 21:52:51 WARN storage.MemoryStore: Not enough space to cache input-0-1442929965600 in memory! (computed 504.0 B so far)
15/09/22 21:52:51 INFO storage.MemoryStore: Memory use = 4.6 MB (blocks) + 529.8 MB (scratch space shared across 1 thread(s)) = 534.4 MB. Storage limit = 534.5 MB.

Root case && resolve:

This is cased by storage MemoryStore is not enough. Two ways to resolve:

Increase spark.storage.memoryFraction, default is 0.6. Could increase to 0.9.
Increase spark.driver.memory, my original is 1g, I increase to 2g. Issue resolved.