Spark内存分析

最新推荐文章于 2024-06-20 01:33:19 发布

桃花源养鸡

最新推荐文章于 2024-06-20 01:33:19 发布

阅读量2.2k

点赞数 17

分类专栏： Spark源码系列文章标签： spark 大数据 big data

本文链接：https://blog.youkuaiyun.com/ralph_wren/article/details/122948905

版权

Spark源码系列专栏收录该内容

3 篇文章

订阅专栏

storage和execution内存的获取

org.apache.spark.memory.UnifiedMemoryManager#getMaxMemory

  /**
   * Return the total amount of memory shared between execution and storage, in bytes.
   */
  private def getMaxMemory(conf: SparkConf): Long = {
    val systemMemory = conf.get(TEST_MEMORY)
    val reservedMemory = conf.getLong(TEST_RESERVED_MEMORY.key,
      if (conf.contains(IS_TESTING)) 0 else RESERVED_SYSTEM_MEMORY_BYTES)
    val minSystemMemory = (reservedMemory * 1.5).ceil.toLong
    if (systemMemory < minSystemMemory) {
      throw new IllegalArgumentException(s"System memory $systemMemory must " +
        s"be at least $minSystemMemory. Please increase heap size using the --driver-memory " +
        s"option or ${config.DRIVER_MEMORY.key} in Spark configuration.")
    }
    // SPARK-12759 Check executor memory to fail fast if memory is insufficient
    if (conf.contains(config.EXECUTOR_MEMORY)) {
      val executorMemory = conf.getSizeAsBytes(config.EXECUTOR_MEMORY.key)
      if (executorMemory < minSystemMemory) {
        throw new IllegalArgumentException(s"Executor memory $executorMemory must be at least " +
          s"$minSystemMemory. Please increase executor memory using the " +
          s"--executor-memory option or ${config.EXECUTOR_MEMORY.key} in Spark configuration.")
      }
    }
    val usableMemory = systemMemory - reservedMemory
    val memoryFraction = conf.get(config.MEMORY_FRACTION)
    (usableMemory * memoryFraction).toLong
  }

在这个方法中,规定了execution和storage能够获得的最大内存.

val usableMemory = systemMemory - reservedMemory
val memoryFraction = conf.get(config.MEMORY_FRACTION)
(usableMemory * memoryFraction).toLong

其中涉及系统保留内存,系统内存,内存比例三个变量

系统保留内存为固定300m

private val RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024

系统总内存由Runtime.getRuntime.maxMemory获取

val TEST_MEMORY = ConfigBuilder("spark.testing.memory")
.version("1.6.0")
.longConf
.createWithDefault(Runtime.getRuntime.maxMemory)

memory fraction默认为0.6

private[spark] val MEMORY_FRACTION = ConfigBuilder("spark.memory.fraction")
.doc("Fraction of (heap space - 300MB) used for execution and storage. The " +
"lower this is, the more frequently spills and cached data eviction occur. " +
"The purpose of this config is to set aside memory for internal metadata, " +
"user data structures, and imprecise size estimation in the case of sparse, " +
"unusually large records. Leaving this at the default value is recommended. ")
.version("1.6.0")
.doubleConf
.createWithDefault(0.6)

executor的内存组成

在org.apache.spark.deploy.yarn.YarnAllocator 中介绍了executor的内存情况

  // Resource capability requested for each executor
  private[yarn] val resource: Resource = {
    val resource: Resource = Resource.newInstance(
      executorMemory + executorOffHeapMemory + memoryOverhead + pysparkWorkerMemory, executorCores)
    ResourceRequestHelper.setResourceRequests(executorResourceRequests, resource)
    logDebug(s"Created resource capability: $resource")
    resource
  }

executor内存由 executorMemory + executorOffHeapMemory + memoryOverhead + pysparkWorkerMemory四部分组成

这四部分来自

  // Executor memory in MiB.
  protected val executorMemory = sparkConf.get(EXECUTOR_MEMORY).toInt
  // Executor offHeap memory in MiB.
  protected val executorOffHeapMemory = YarnSparkHadoopUtil.executorOffHeapMemorySizeAsMb(sparkConf)
  // Additional memory overhead.
  protected val memoryOverhead: Int = sparkConf.get(EXECUTOR_MEMORY_OVERHEAD).getOrElse(
    math.max((MEMORY_OVERHEAD_FACTOR * executorMemory).toInt, MEMORY_OVERHEAD_MIN)).toInt
  protected val pysparkWorkerMemory: Int = if (sparkConf.get(IS_PYTHON_APP)) {
    sparkConf.get(PYSPARK_EXECUTOR_MEMORY).map(_.toInt).getOrElse(0)
  } else {
    0
  }

对应的几个参数分别为

executor内存

  private[spark] val EXECUTOR_MEMORY = ConfigBuilder(SparkLauncher.EXECUTOR_MEMORY)
    .doc("Amount of memory to use per executor process, in MiB unless otherwise specified.")
    .version("0.7.0")
    .bytesConf(ByteUnit.MiB)
    .createWithDefaultString("1g")

executor堆外内存,先判定是否开启,再检查设置的对外内存是否大于0

def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf): Int = {
    if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
      val sizeInMB = Utils.memoryStringToMb(sparkConf.get(MEMORY_OFFHEAP_SIZE).toString)
      require(sizeInMB > 0,
        s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when ${MEMORY_OFFHEAP_ENABLED.key} == true")
      sizeInMB
    } else {
      0
    }
  }

  private[spark] val EXECUTOR_MEMORY_OVERHEAD = ConfigBuilder("spark.executor.memoryOverhead")
    .doc("The amount of non-heap memory to be allocated per executor, in MiB unless otherwise" +
      " specified.")
    .version("2.3.0")
    .bytesConf(ByteUnit.MiB)
    .createOptional

第四个python的不考虑

在本地启动spark-shell验证设置的内存是否符合以上规定

设置driver内存为2g

spark-shell --conf spark.driver.memory=2g

获取对应变量

//获取driver内存,在本地模式driver充当一个executor
scala> sc.getConf.getSizeAsBytes("spark.driver.memory")
res0: Long = 2147483648

//获取系统内存
scala> val systemMemory = Runtime.getRuntime.maxMemory
systemMemory: Long = 1908932608

//固定系统保留内存300m
scala> val reservedMemory = 300 * 1024 * 1024
reservedMemory: Int = 314572800

//内存比例默认0.6
scala> val memoryFraction = sc.getConf.getDouble("spark.memory.fraction", 0.6)
memoryFraction: Double = 0.6

//系统内存减去保留内存
scala> val usableMemory = systemMemory - reservedMemory
usableMemory: Long = 1594359808

//根据公式获取最大内存
scala> val maxMemory = (usableMemory * memoryFraction).toLong
maxMemory: Long = 956615884

//将最大内存转换为M
scala> 956615884/1024/1024d
res6: Double = 912.2998046875

在spark管理页面看到storage memory确实为912.3m,符合计算结果

参考文档 apache spark - How does web UI calculate Storage Memory (in Executors tab)? - Stack Overflow

同时,发现 Runtime.getRuntime.maxMemory 获得的内存大小小于设置的2g

scala> 1908932608/1024/1024d
res3: Double = 1820.5

这是因为在虚拟机中分为老年代和年轻代,而在年轻代中有一个Eden区和两个Survivor区,而同时只有一个survivor是可用的,因此造成可用内存大小小于总内存大小

再启动一个spark-submit任务,设置打印gc日志

spark-submit --class com.getui.bigdata.spark_study.ReadData \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 1g \
    --executor-memory 2g \
    --executor-cores 1 \
--conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC" \
--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=/Users/mac/IdeaProjects/spark/conf/log4j.properties" \
/Users/mac/IdeaProjects/namespace/spark/target/spark-0.1-independency.jar

{Heap before GC invocations=6 (full 1):
 PSYoungGen      total 105984K, used 105830K [0x0000000795580000, 0x000000079e500000, 0x00000007c0000000)
  eden space 100864K, 100% used [0x0000000795580000,0x000000079b800000,0x000000079b800000)
  from space 5120K, 96% used [0x000000079e000000,0x000000079e4d98f0,0x000000079e500000)
  to   space 10752K, 0% used [0x000000079d000000,0x000000079d000000,0x000000079da80000)
 ParOldGen       total 100864K, used 6714K [0x0000000740000000, 0x0000000746280000, 0x0000000795580000)
  object space 100864K, 6% used [0x0000000740000000,0x000000074068ea60,0x0000000746280000)
 Metaspace       used 32269K, capacity 33870K, committed 34176K, reserved 1079296K
  class space    used 4426K, capacity 4625K, committed 4736K, reserved 1048576K
10.143: [GC (Allocation Failure) [PSYoungGen: 105830K->7867K(136192K)] 112544K->14590K(237056K), 0.0078949 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
Heap after GC invocations=6 (full 1):
 PSYoungGen      total 136192K, used 7867K [0x0000000795580000, 0x000000079fb80000, 0x00000007c0000000)
  eden space 125440K, 0% used [0x0000000795580000,0x0000000795580000,0x000000079d000000)
  from space 10752K, 73% used [0x000000079d000000,0x000000079d7aeef8,0x000000079da80000)
  to   space 10752K, 0% used [0x000000079f100000,0x000000079f100000,0x000000079fb80000)
 ParOldGen       total 100864K, used 6722K [0x0000000740000000, 0x0000000746280000, 0x0000000795580000)
  object space 100864K, 6% used [0x0000000740000000,0x0000000740690a60,0x0000000746280000)
 Metaspace       used 32269K, capacity 33870K, committed 34176K, reserved 1079296K
  class space    used 4426K, capacity 4625K, committed 4736K, reserved 1048576K
}

7.157: [Full GC (Metadata GC Threshold) [PSYoungGen: 6495K->0K(240640K)] [ParOldGen: 12872K->13889K(184320K)] 19368K->13889K(424960K), [Metaspace: 54854K->54854K(1101824K)], 0.0745744 secs] [Times: user=0.37 sys=0.01, real=0.08 secs]