10.BlockManager源码分析

最新推荐文章于 2024-06-07 20:24:47 发布
原创 最新推荐文章于 2024-06-07 20:24:47 发布 · 904 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文深入解析了Apache Spark中BlockManager的实现原理，包括BlockManager的注册流程、数据的读写机制以及BlockManagerMasterActor如何维护元数据。同时，文章详细介绍了BlockManager在不同存储层级上的数据管理策略。

先来张原理图吧 !



上面的BlockManagerWorker是在1.3之前的版本中才有的 , 从1.3开始BlockManagerWorker由BlockTransferService替代 . 
 
其实BlockManagerMaster的主要工作是交于BlockManagerMasterActor来处理的:
  1. /**
  2.  * BlockManagerMasterActor is an actor on the master node to track statuses of
  3.  * all slaves' block managers.
  4.  * 
  5.  * 其实BlockManagerMasterActor就是负责维护各个executor的BlockManager的元数据
  6.  * BlockManagerInfo , BlockStatus
  7.  */
  8. private[spark]
  9. class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus: LiveListenerBus)
  10.   extends Actor with ActorLogReceive with Logging {

该类中有一个内部类就是BlockManagerInfo , 负责存放每一BlockManager的信息并更新和删除block状态的改变:
  1. /**
  2.  * 每一个BlockManager的BlockManagerInfo
  3.  */
  4. private[spark] class BlockManagerInfo(
  5.     val blockManagerId: BlockManagerId,
  6.     timeMs: Long,
  7.     val maxMem: Long,
  8.     val slaveActor: ActorRef)
  9.   extends Logging {
  11.   private var _lastSeenMs: Long = timeMs
  12.   private var _remainingMem: Long = maxMem
  14.   // Mapping from block id to its status.
  15.   private val _blocks = new JHashMap[BlockId, BlockStatus]
  17.   def getStatus(blockId: BlockId) = Option(_blocks.get(blockId))
  19.   def updateLastSeenMs() {
  20.     _lastSeenMs = System.currentTimeMillis()
  21.   }
  23.   def updateBlockInfo(
  24.       blockId: BlockId,
  25.       storageLevel: StorageLevel,
  26.       memSize: Long,
  27.       diskSize: Long,
  28.       tachyonSize: Long) {
  30.     updateLastSeenMs()
  32.     /**
  33.      * 判断如果内部有这个block
  34.      */
  35.     if (_blocks.containsKey(blockId)) {
  36.       // The block exists on the slave already.
  37.       val blockStatus: BlockStatus = _blocks.get(blockId)
  38.       val originalLevel: StorageLevel = blockStatus.storageLevel
  39.       val originalMemSize: Long = blockStatus.memSize
  41.       // 判断如果storagelevel是使用内存那么就给剩余内存数量加上当前的内存量
  42.       if (originalLevel.useMemory) {
  43.         _remainingMem += originalMemSize
  44.       }
  45.     }
  47.     // 给Block创建一份BlockStatus,然后根据其持久化级别对相应的内存资源进行计算
  48.     if (storageLevel.isValid) {
  49.       /* isValid means it is either stored in-memory, on-disk or on-Tachyon.
  50.        * The memSize here indicates the data size in or dropped from memory,
  51.        * tachyonSize here indicates the data size in or dropped from Tachyon,
  52.        * and the diskSize here indicates the data size in or dropped to disk.
  53.        * They can be both larger than 0, when a block is dropped from memory to disk.
  54.        * Therefore, a safe way to set BlockStatus is to set its info in accurate modes. */
  55.       if (storageLevel.useMemory) {
  56.         _blocks.put(blockId, BlockStatus(storageLevel, memSize, 0, 0))
  57.         _remainingMem -= memSize
  58.         logInfo("Added %s in memory on %s (size: %s, free: %s)".format(
  59.           blockId, blockManagerId.hostPort, Utils.bytesToString(memSize),
  60.           Utils.bytesToString(_remainingMem)))
  61.       }
  62.       if (storageLevel.useDisk) {
  63.         _blocks.put(blockId, BlockStatus(storageLevel, 0, diskSize, 0))
  64.         logInfo("Added %s on disk on %s (size: %s)".format(
  65.           blockId, blockManagerId.hostPort, Utils.bytesToString(diskSize)))
  66.       }
  67.       if (storageLevel.useOffHeap) {
  68.         _blocks.put(blockId, BlockStatus(storageLevel, 0, 0, tachyonSize))
  69.         logInfo("Added %s on tachyon on %s (size: %s)".format(
  70.           blockId, blockManagerId.hostPort, Utils.bytesToString(tachyonSize)))
  71.       }
  72.       
  73.       // 如果StorageLevel是非法的而且之前保存过这个blockId那么就将blockId从内存中删除
  74.     } else if (_blocks.containsKey(blockId)) {
  75.       // If isValid is not true, drop the block.
  76.       val blockStatus: BlockStatus = _blocks.get(blockId)
  77.       _blocks.remove(blockId)
  78.       if (blockStatus.storageLevel.useMemory) {
  79.         logInfo("Removed %s on %s in memory (size: %s, free: %s)".format(
  80.           blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.memSize),
  81.           Utils.bytesToString(_remainingMem)))
  82.       }
  83.       if (blockStatus.storageLevel.useDisk) {
  84.         logInfo("Removed %s on %s on disk (size: %s)".format(
  85.           blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.diskSize)))
  86.       }
  87.       if (blockStatus.storageLevel.useOffHeap) {
  88.         logInfo("Removed %s on %s on tachyon (size: %s)".format(
  89.           blockId, blockManagerId.hostPort, Utils.bytesToString(blockStatus.tachyonSize)))
  90.       }
  91.     }
  92.   }

每一个BlockManager刚开始的时候都会向BlockManagerMasterActor发送 "RegisterBlockManager"消息进行注册 , BlockManagerMasterActor接收到消息之后的注册代码如下:
  1. /**
  2.    * 接收消息处理的方法
  3.    */
  4.   override def receiveWithLogging = {
  5.     /**
  6.      * 首先BlockManagerMaster需要接收到其它节点的BlockManager注册的消息
  7.      */
  8.     case RegisterBlockManager(blockManagerId, maxMemSize, slaveActor) =>
  9.       // 调用注册方法
  10.       register(blockManagerId, maxMemSize, slaveActor)
  11.       // 发送注册成功的消息
  12.       sender ! true
  14.     /**
  15.      * 发送Block信息更改
  16.      */
  17.     case UpdateBlockInfo(
  18.       blockManagerId, blockId, storageLevel, deserializedSize, size, tachyonSize) =>
  19.       sender ! updateBlockInfo(
  20.         blockManagerId, blockId, storageLevel, deserializedSize, size, tachyonSize)

里面会调用register方法 , Block状态更改的消息下面会分析 , 源码如下:
  1. /**
  2.    * BlockManager注册的方法
  3.    */
  4.   private def register(id: BlockManagerId, maxMemSize: Long, slaveActor: ActorRef) {
  5.     val time = System.currentTimeMillis()
  6.     // 首先判断一下HashMap中没有指定的BlockManagerId, 说明从来没有注册过那么才会继续往下走去注册这个BlockManager
  7.     if (!blockManagerInfo.contains(id)) {
  8.       
  9.       // 根据BlockManager对应的ExecutorId找到对应的BlockManagerInfo
  10.       // 做一个安全判断 , 如果没有BlockManagerId那么同步到blockManagerIByExecutorId里面
  11.       // 如果BlockManagerIdByExecutor有的话就做一下清理
  12.       blockManagerIdByExecutor.get(id.executorId) match {
  13.         case Some(oldId) =>
  14.           // A block manager of the same executor already exists, so remove it (assumed dead)
  15.           logError("Got two different block manager registrations on same executor - " 
  16.               + s" will replace old one $oldId with new one $id")
  17.           // 从内存中移除掉executorId相关的blockManagerInfo
  18.           removeExecutor(id.executorId)  
  19.         case None =>
  20.       }
  21.       logInfo("Registering block manager %s with %s RAM, %s".format(
  22.         id.hostPort, Utils.bytesToString(maxMemSize), id))
  23.       
  24.       // 往blockmanagerIdByExecutor map 中保存一份executorid到blockmanagerId的映射
  25.       blockManagerIdByExecutor(id.executorId) = id
  26.       
  27.       // 为BlockManagerId创建一份BlockManagerInfo
  28.       // 并往BlockManagerInfo map中保存一份blockmanagerId到blockmanagerInfo的映射
  29.       // 到这里注册BlockManager就完咯
  30.       blockManagerInfo(id) = new BlockManagerInfo(
  31.         id, System.currentTimeMillis(), maxMemSize, slaveActor)
  32.     }
  33.     listenerBus.post(SparkListenerBlockManagerAdded(time, id, maxMemSize))
  34.   }
其实BlockManager的注册就是将BlockManagerId和与这个BlockManager相关联的ExecutorId加入BlockManagerInfo和BlockManagerIdByExecutor中 ,下面的BlockManagerMasterActor的三个成员变量负责存放这些信息:
  1. // 首先这个map 映射了block manager id 到block manager info 之间的映射关系
  2.   // BlockManagerMaster要负责维护每个BlockManager的BlockManagerInfo
  3.   // 而Spark中管理数据的最小单位为Block , 同Hadoop一样
  4.   private val blockManagerInfo = new mutable.HashMap[BlockManagerId, BlockManagerInfo]
  6.   // Mapping from executor ID to block manager ID.
  7.   private val blockManagerIdByExecutor = new mutable.HashMap[String, BlockManagerId]
  9.   // Mapping from block id to the set of block managers that have the block.
  10.   private val blockLocations = new JHashMap[BlockId, mutable.HashSet[BlockManagerId]]


接下来就分析一下BlockManager的源码 , BlockManager运行在每个节点上,包括driver和executor都会有一份,主要提供关联在本地或者远程存取数据的功能 , 并支持内存,磁盘和对外存储
,该类中有一成员变量会对每一个block信息进行缓存,如下:
  1. // 这里还有一个东西就是每个BlockManager自己会维护一个map在内存中存放一个一个的block块
  2.   // 这个块是blockId到blcokInfo的映射
  3.   // 每个BlockInfo中是不是就封装了Block的数据
  4.   // BlockInfo最大的作用是用于作为多线程访问同一个Block的同步监视器
  5.   private val blockInfo = new TimeStampedHashMap[BlockId, BlockInfo]

最先调用BlockManager的initialize方法对一些组件进行初始化:
  1. def initialize(appId: String): Unit = {
  2.     
  3.     // 在1.3版本之前BlockManager是使用BlockManagerWorker来进行通信的 , 但1.3版本已经改为BlockTransfer
  4.     // 所以首先初始化用于block数据传输的BlockTransferService
  5.     blockTransferService.init(this)
  6.     shuffleClient.init(appId)
  8.     //为当前这个BlockManager创建一个唯一的BlockManagerId,使用到了executorId(每个BlockManager都关联一个executor,BlockTransferService的hostname以及port)
  9.     // 所以 , 从这个BlockManagerId的初始化即可看出一个BlockManager是通过一个节点上的executor来唯一标识的
  10.     blockManagerId = BlockManagerId(
  11.       executorId, blockTransferService.hostName, blockTransferService.port)
  13.     shuffleServerId = if (externalShuffleServiceEnabled) {
  14.       BlockManagerId(executorId, blockTransferService.hostName, externalShuffleServicePort)
  15.     } else {
  16.       blockManagerId
  17.     }
  19.     // 使用BlockManagerMasterActor引用进行BlockManager的注册 , 发送消息到BlockManagerMasterActor
  20.     master.registerBlockManager(blockManagerId, maxMemory, slaveActor)
  22.     // Register Executors' configuration with the local shuffle service, if one should exist.
  23.     if (externalShuffleServiceEnabled && !blockManagerId.isDriver) {
  24.       registerWithExternalShuffleServer()
  25.     }
  26.   }

上面的代码中调用registerBlockManager方法使用BlockManagerMasterActor的引用进行BlockManager进行注册,发送消息到BlockManagerMasterActor , 后面就是调用
BlockManagerMasterActor 的register方法了 , 上面已经提到过.


接下来就是数据的获取了 , 分别是从本地获取数据和从远程的节点上获取数据 , 以下是BlockManager从本地获取数据的源码:
  1. /**
  2.    * 从本地获取数据
  3.    */
  4.   private def doGetLocal(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
  5.     // 首先尝试直接从内存中获取数据
  6.     val info = blockInfo.get(blockId).orNull
  7.     if (info != null) {
  8.       // 对所有的BlockInfo都会进行多线程并发访问的同步操作
  9.       // 所有BlockInfo相当于是对一个Block用于作为多线程并发访问的同步监视器
  10.       info.synchronized {
  11.         // Double check to make sure the block is still there. There is a small chance that the
  12.         // block has been removed by removeBlock (which also synchronizes on the blockInfo object).
  13.         // Note that this only checks metadata tracking. If user intentionally deleted the block
  14.         // on disk or from off heap storage without using removeBlock, this conditional check will
  15.         // still pass but eventually we will get an exception because we can't find the block.
  16.         if (blockInfo.get(blockId).isEmpty) {
  17.           logWarning(s"Block $blockId had been removed")
  18.           return None
  19.         }
  21.         // If another thread is writing the block, wait for it to become ready.
  22.         // 如果其他线程在操作这个block那么其实会卡住等待后去BlockInfo的排他锁
  23.         // 如果始终没有获取到则返回false
  24.         if (!info.waitForReady()) {
  25.           // If we get here, the block write failed.
  26.           logWarning(s"Block $blockId was marked as failure.")
  27.           return None
  28.         }
  30.         val level = info.level
  31.         logDebug(s"Level for block $blockId is $level")
  33.         // Look for the block in memory
  34.         // 判断如果持久化级别使用了内存比如MEMORY_ONLY,MEMORY_AND_DISK_SER
  35.         // 尝试从MemoryStore中获取数据
  36.         if (level.useMemory) {
  37.           logDebug(s"Getting block $blockId from memory")
  38.           val result = if (asBlockResult) {
  39.             memoryStore.getValues( ).map(new BlockResult(_, DataReadMethod.Memory, info.size))
  40.           } else {
  41.             memoryStore.getBytes(blockId)
  42.           }
  43.           result match {
  44.             case Some(values) =>
  45.               return result
  46.             case None =>
  47.               logDebug(s"Block $blockId not found in memory")
  48.           }
  49.         }
  51.         // Look for the block in Tachyon
  52.         if (level.useOffHeap) {
  53.           logDebug(s"Getting block $blockId from tachyon")
  54.           if (tachyonStore.contains(blockId)) {
  55.             tachyonStore.getBytes(blockId) match {
  56.               case Some(bytes) =>
  57.                 if (!asBlockResult) {
  58.                   return Some(bytes)
  59.                 } else {
  60.                   return Some(new BlockResult(
  61.                     dataDeserialize(blockId, bytes), DataReadMethod.Memory, info.size))
  62.                 }
  63.               case None =>
  64.                 logDebug(s"Block $blockId not found in tachyon")
  65.             }
  66.           }
  67.         }
  69.         // Look for block on disk, potentially storing it back in memory if required
  70.         // 判断如果持久化级别使用了硬盘持久化
  71.         // 尝试从DiskStore中获取数据
  72.         if (level.useDisk) {
  73.           logDebug(s"Getting block $blockId from disk")
  74.           // 通过DiskStore的getBytes获取数据
  75.           val bytes: ByteBuffer = diskStore.getBytes(blockId) match {
  76.             case Some(b) => b
  77.             case None =>
  78.               throw new BlockException(
  79.                 blockId, s"Block $blockId not found on disk, though it should be")
  80.           }
  81.           assert(0 == bytes.position())
  83.           // 如果数据仅仅只是设置了硬盘持久化而没有设置没存持久化存储那么直接将数据封装在BlockResult中返回
  84.           if (!level.useMemory) {
  85.             // If the block shouldn't be stored in memory, we can just return it
  86.             if (asBlockResult) {
  87.               return Some(new BlockResult(dataDeserialize(blockId, bytes), DataReadMethod.Disk,
  88.                 info.size))
  89.             } else {
  90.               return Some(bytes)
  91.             }
  92.           } else {
  93.             // Otherwise, we also have to store something in the memory store
  94.             
  95.         	  // 如果数据还设置了内存持久化那么需要将数据通过memoryStore放入内存中
  96.             if (!level.deserialized || !asBlockResult) {
  97.               /* We'll store the bytes in memory if the block's storage level includes
  98.                * "memory serialized", or if it should be cached as objects in memory
  99.                * but we only requested its serialized bytes. */
  100.               val copyForMemory = ByteBuffer.allocate(bytes.limit)
  101.               copyForMemory.put(bytes)
  102.               memoryStore.putBytes(blockId, copyForMemory, level)
  103.               bytes.rewind()
  104.             }
  105.             if (!asBlockResult) {
  106.               return Some(bytes)
  107.             } else {
  108.               val values = dataDeserialize(blockId, bytes)
  109.               if (level.deserialized) {
  110.                 // Cache the values before returning them
  111.                 val putResult = memoryStore.putIterator(
  112.                   blockId, values, level, returnValues = true, allowPersistToDisk = false)
  113.                 // The put may or may not have succeeded, depending on whether there was enough
  114.                 // space to unroll the block. Either way, the put here should return an iterator.
  115.                 putResult.data match {
  116.                   case Left(it) =>
  117.                     return Some(new BlockResult(it, DataReadMethod.Disk, info.size))
  118.                   case _ =>
  119.                     // This only happens if we dropped the values back to disk (which is never)
  120.                     throw new SparkException("Memory store did not return an iterator!")
  121.                 }
  122.               } else {
  123.                 return Some(new BlockResult(values, DataReadMethod.Disk, info.size))
  124.               }
  125.             }
  126.           }
  127.         }
  128.       }
  129.     } else {
  130.       logDebug(s"Block $blockId not registered locally")
  131.     }
  132.     None
  133.   }

代码有点多 , 其实总结下来就两大块 , 从硬盘上获取数据和从内存中获取数据 , 在拿取数据的时候必须是线程安全的 , 防止多个BlockManager来读取同一份数据 , 
首先从内存中获取 , 判断数据是否被内存持久化 , 然后调用MemoryStore的getValues或者getBytes方法获取数据 , 至于调用哪个方法看返回的结果是否为BlockResult , 
而数据若是被硬盘持久化的那么就用DiskStore的getBytes方法获取 , 获取到之后在检查一下是否需要内存持久化 , 是的话还得调用MemoryStore存储在内存中 , 一下是MemoryStore和DiskStore的数据获取方法getVlaues和getBytes :
MemoryStore的:
  1. // MemoryStore中维护的entries map中其实就是真的存放的是每个block的数据了
  2.   // 每个block在内存中的数据用MemoryEntry代表
  3.   private val entries = new LinkedHashMap[BlockId, MemoryEntry](32, 0.75f, true)
最后其实block的数据其实是存储在一个类型为HashMap的entries成员变量中了
  1. override def getBytes(blockId: BlockId): Option[ByteBuffer] = {
  2.     // entries也是多线程并发访问同步的
  3.     val entry = entries.synchronized {
  4.       // 尝试从内存中获取block数据
  5.       entries.get(blockId)
  6.     }
  7.     // 如果没有获取到范湖None
  8.     if (entry == null) {
  9.       None
  10.     // 如果获取到了非序列化的数据
  11.     } else if (entry.deserialized) {
  12.       // 调用BlockManager的数据序列化方法将数据序列化返回
  13.       Some(blockManager.dataSerialize(blockId, entry.value.asInstanceOf[Array[Any]].iterator))
  14.     } else {
  15.       // 否则直接返回数据
  16.       Some(entry.value.asInstanceOf[ByteBuffer].duplicate()) // Doesn't actually copy the data
  17.     }
  18.   }
  20.   override def getValues(blockId: BlockId): Option[Iterator[Any]] = {
  21.     val entry = entries.synchronized {
  22.       entries.get(blockId)
  23.     }
  24.     if (entry == null) {
  25.       None
  26.      // 如果非序列化直接返回
  27.     } else if (entry.deserialized) {
  28.       Some(entry.value.asInstanceOf[Array[Any]].iterator)
  29.       // 如果序列化了那么用BlockManager进行反序列化再返回
  30.     } else {
  31.       val buffer = entry.value.asInstanceOf[ByteBuffer].duplicate() // Doesn't actually copy data
  32.       Some(blockManager.dataDeserialize(blockId, buffer))
  33.     }
  34.   }
DiskStore的:
  1. private def getBytes(file: File, offset: Long, length: Long): Option[ByteBuffer] = {
  2.     // DiskStore底层使用的是java的nio进行文件的读写操作
  3.     val channel = new RandomAccessFile(file, "r").getChannel
  5.     try {
  6.       // For small files, directly read rather than memory map
  7.       if (length < minMemoryMapBytes) {
  8.         val buf = ByteBuffer.allocate(length.toInt)
  9.         channel.position(offset)
  10.         while (buf.remaining() != 0) {
  11.           if (channel.read(buf) == -1) {
  12.             throw new IOException("Reached EOF before filling buffer\n" +
  13.               s"offset=$offset\nfile=${file.getAbsolutePath}\nbuf.remaining=${buf.remaining}")
  14.           }
  15.         }
  16.         buf.flip()
  17.         Some(buf)
  18.       } else {
  19.         Some(channel.map(MapMode.READ_ONLY, offset, length))
  20.       }
  21.     } finally {
  22.       channel.close()
  23.     }
  24.   }

接下来是从别的节点获取数据:
  1. /**
  2.    * 从别的节点拿取数据
  3.    */
  4.   private def doGetRemote(blockId: BlockId, asBlockResult: Boolean): Option[Any] = {
  5.     require(blockId != null, "BlockId is null")
  6.     // 首先BlockManagerMaster上获取每个blockId对应的BlockManager的信息
  7.     // 然后打乱
  8.     val locations = Random.shuffle(master.getLocations(blockId))
  9.     // 遍历每一个BlockManager
  10.     for (loc <- locations) {
  11.       logDebug(s"Getting remote block $blockId from $loc")
  12.       // 使用BlockTransferService进行异步的远程网络获取 , 将block数据传输回来
  13.       // 连接的时候使用的BlockManager的唯一标识,就是host,port,executorId
  14.       val data = blockTransferService.fetchBlockSync(
  15.         loc.host, loc.port, loc.executorId, blockId.toString).nioByteBuffer()
  17.       if (data != null) {
  18.         if (asBlockResult) {
  19.           return Some(new BlockResult(
  20.             dataDeserialize(blockId, data),
  21.             DataReadMethod.Network,
  22.             data.limit()))
  23.         } else {
  24.           return Some(data)
  25.         }
  26.       }
  27.       logDebug(s"The value of block $blockId is null")
  28.     }
  29.     logDebug(s"Block $blockId not found")
  30.     None
  31.   }
上面的master.getLocations(blockId)获取到的数据是在BlockManagerMasterActor中的一个成员变量 , 在获取到这些有数据的BlockManagerId之后在进行遍历拉取数据:
  1. // 这个变量存储了一个blockId代表的数据在别的节点上有哪些BlockManager拥有 , 若是做了数据的备份那么本地中一个blockId对应这别的节点上的多个BlockManagerId
  2.   private val blockLocations = new JHashMap[BlockId, mutable.HashSet[BlockManagerId]]
然后通过blockTransferService的fetchBlockSync异步将数据拉取过来 , 后面再做一些反序列化的操作 , 这就是doGetRemote从远程获取数据 . 


那么既然有读数据那么就有写数据 , 有三步骤:
1.写内存不足的处理机制是什么? -> 先将旧的数据硬盘持久化 , 将新的数据放入内存 , 若还是不行的话那么就将新的数据硬盘持久化
2.写完以后汇报BlockManagerMasterActor
3.如果要复用的话随机挑一个BlockManager通过BlockTransferService将数据传输过去


1.写内存不足的处理机制是什么?
先来看看BlockManager中的doPut()方法的部分源码(因为有点多):
  1. // 为需要进行存储的block创建一个BlockInfo对象并放入blockInfo map中
  2.     val putBlockInfo = {
  3.       val tinfo = new BlockInfo(level, tellMaster)
  4.       // Do atomically !
  5.       val oldBlockOpt = blockInfo.putIfAbsent(blockId, tinfo)
  6.       if (oldBlockOpt.isDefined) {
  7.         if (oldBlockOpt.get.waitForReady()) {
  8.           logWarning(s"Block $blockId already exists on this machine; not re-adding it")
  9.           return updatedBlocks
  10.         }
  11.         // TODO: So the block info exists - but previous attempt to load it (?) failed.
  12.         // What do we do now ? Retry on it ?
  13.         oldBlockOpt.get
  14.       } else {
  15.         tinfo
  16.       }
  17.     }
首先创建一个putBlockInfo 函数 , 在函数中创建BlockInfo对象

然后接下来是对putBlockInfo函数进行枷锁操作:
  1. // 尝试对blockInfo枷锁,进行多线程并发访问同步 
  2.     putBlockInfo.synchronized {
  3.       logTrace("Put for block %s took %s to get into synchronized block"
  4.         .format(blockId, Utils.getUsedTimeMs(startTimeMs)))
  6.       var marked = false
  7.       try {
  8.         // returnValues - Whether to return the values put
  9.         // blockStore - The type of storage to put these values into
  10.         
  11.         // 首先根据持久化级别选择一种blockStore,memroyStore,diskStore等
  12.         val (returnValues, blockStore: BlockStore) = {
  13.           if (putLevel.useMemory) {
  14.             // Put it in memory first, even if it also has useDisk set to true;
  15.             // We will drop it to disk later if the memory store can't hold it.
  16.             (true, memoryStore)
  17.           } else if (putLevel.useOffHeap) {
  18.             // Use tachyon for off-heap storage
  19.             (false, tachyonStore)
  20.           } else if (putLevel.useDisk) {
  21.             // Don't get back the bytes from put unless we replicate them
  22.             (putLevel.replication > 1, diskStore)
  23.           } else {
  24.             assert(putLevel == StorageLevel.NONE)
  25.             throw new BlockException(
  26.               blockId, s"Attempted to put block $blockId without specifying storage level!")
  27.           }
  28.         }
  30.         // Actually put the values
  31.         
  32.         // 根据你选择的store , 根据数据的类型将数据放入store中 , 要么是MemoryStore的putBytes要么是DiskStore的putBytes
  33.         val result = data match {
  34.           case IteratorValues(iterator) =>
  35.             blockStore.putIterator(blockId, iterator, putLevel, returnValues)
  36.           case ArrayValues(array) =>
  37.             blockStore.putArray(blockId, array, putLevel, returnValues)
  38.           case ByteBufferValues(bytes) =>
  39.             bytes.rewind()
  40.             blockStore.putBytes(blockId, bytes, putLevel)
  41.         }
  42.         size = result.size
  43.         result.data match {
  44.           case Left (newIterator) if putLevel.useMemory => valuesAfterPut = newIterator
  45.           case Right (newBytes) => bytesAfterPut = newBytes
  46.           case _ =>
  47.         }
  49.         // Keep track of which blocks are dropped from memory
  50.         if (putLevel.useMemory) {
  51.           result.droppedBlocks.foreach { updatedBlocks += _ }
  52.         }
  54.         // putBlockInfo作为一个函数参数放入getCurrentBlockStatus获取到一个block对应的BlockStatus,putBlockInfo函数的方法会执行
  55.         val putBlockStatus = getCurrentBlockStatus(blockId, putBlockInfo)
  56.         if (putBlockStatus.storageLevel != StorageLevel.NONE) {
  57.           // Now that the block is in either the memory, tachyon, or disk store,
  58.           // let other threads read it, and tell the master about it.
  59.           marked = true
  60.           putBlockInfo.markReady(size)
  61.           if (tellMaster) {
  62.             // 调用reportBlockStatus()方法将新写入的block数据发送到BlockManagerMasterActor以便于进行block元数据的同步和维护
  63.             reportBlockStatus(blockId, putBlockInfo, putBlockStatus)
  64.           }
  65.           updatedBlocks += ((blockId, putBlockStatus))
  66.         }
  67.       }
 加锁的代码中需要根据持久化级别选择存储方式 , 根据不同的存储方式调用MemeoryStore的不同存储方法 , 这里详细看一下存储的具体代码:
MemoryStore中不管是putArray或者是putBytes方法都会调用tryToPut将数据优先放入内存,不行的话则尝试移除部分旧数据再将block存入:
  1. /**
  2.    * 优先放入内存 , 不行的话尝试移除部分旧数据再讲block存入 
  3.    */
  4.   private def tryToPut(
  5.       blockId: BlockId,
  6.       value: Any,
  7.       size: Long,
  8.       deserialized: Boolean): ResultWithDroppedBlocks = {
  10.     /* TODO: Its possible to optimize the locking by locking entries only when selecting blocks
  11.      * to be dropped. Once the to-be-dropped blocks have been selected, and lock on entries has
  12.      * been released, it must be ensured that those to-be-dropped blocks are not double counted
  13.      * for freeing up more space for another block that needs to be put. Only then the actually
  14.      * dropping of blocks (and writing to disk if necessary) can proceed in parallel. */
  16.     var putSuccess = false
  17.     val droppedBlocks = new ArrayBuffer[(BlockId, BlockStatus)]
  19.     // 这里必须进行多线程并发同步
  20.     // 要是不这样操作的话当你刚判定内存足够放数据的时候但是其它线程也在放那么就OOM了
  21.     accountingLock.synchronized {
  22.       
  23.       // 调用ensureFreeSpace方法判断内存是否够用,如果不够用此时会将部分数据用dropFromMemory()方法尝试写入磁盘,但是如果持久化不支持磁盘那么数据丢失
  24.       val freeSpaceResult = ensureFreeSpace(blockId, size)
  25.       val enoughFreeSpace = freeSpaceResult.success
  26.       droppedBlocks ++= freeSpaceResult.droppedBlocks
  28.       // 将数据写入内存的时候首先调用enoughFreeSpace()方法 , 判断内存是否够放入数据
  29.       if (enoughFreeSpace) {
  30.         val entry = new MemoryEntry(value, size, deserialized)
  31.         entries.synchronized {
  32.           entries.put(blockId, entry)
  33.           currentMemory += size
  34.         }
  35.         val valuesOrBytes = if (deserialized) "values" else "bytes"
  36.         logInfo("Block %s stored as %s in memory (estimated size %s, free %s)".format(
  37.           blockId, valuesOrBytes, Utils.bytesToString(size), Utils.bytesToString(freeMemory)))
  38.         putSuccess = true
  39.       } else {
  40.         // Tell the block manager that we couldn't put it in memory so that it can drop it to
  41.         // disk if the block allows disk storage.
  42.         val data = if (deserialized) {
  43.           Left(value.asInstanceOf[Array[Any]])
  44.         } else {
  45.           Right(value.asInstanceOf[ByteBuffer].duplicate())
  46.         }
  47.         // 调用dropFromMemory尝试将数据写入磁盘,但是如果block的持久化级别没有说可以写入磁盘那么数据就彻底丢啦
  48.         val droppedBlockStatus = blockManager.dropFromMemory(blockId, data)
  49.         droppedBlockStatus.foreach { status => droppedBlocks += ((blockId, status)) }
  50.       }
  51.     }
  52.     ResultWithDroppedBlocks(putSuccess, droppedBlocks)
  53.   }
在做数据存入的时候肯定需要判断内存是否够用 , ensureFreeSpace方法就是这个作用 , 源码如下:
  1.     blockIdToAdd: BlockId,
  2.       space: Long): ResultWithDroppedBlocks = {
  3.     logInfo(s"ensureFreeSpace($space) called with curMem=$currentMemory, maxMem=$maxMemory")
  5.     val droppedBlocks = new ArrayBuffer[(BlockId, BlockStatus)]
  7.     if (space > maxMemory) {
  8.       logInfo(s"Will not store $blockIdToAdd as it is larger than our memory limit")
  9.       return ResultWithDroppedBlocks(success = false, droppedBlocks)
  10.     }
  12.     // Take into account the amount of memory currently occupied by unrolling blocks
  13.     val actualFreeMemory = freeMemory - currentUnrollMemory
  15.     // 如果当前内存不足够将这个block放入的话
  16.     if (actualFreeMemory < space) {
  17.       val rddToAdd = getRddId(blockIdToAdd)
  18.       val selectedBlocks = new ArrayBuffer[BlockId]
  19.       var selectedMemory = 0L
  21.       // This is synchronized to ensure that the set of entries is not changed
  22.       // (because of getValue or getBytes) while traversing the iterator, as that
  23.       // can lead to exceptions.
  24.       // 同步entries
  25.       entries.synchronized {
  26.         val iterator = entries.entrySet().iterator()
  27.         // 尝试从entries中移除一部分数据
  28.         while (actualFreeMemory + selectedMemory < space && iterator.hasNext) {
  29.           val pair = iterator.next()
  30.           val blockId = pair.getKey
  31.           if (rddToAdd.isEmpty || rddToAdd != getRddId(blockId)) {
  32.             selectedBlocks += blockId
  33.             selectedMemory += pair.getValue.size
  34.           }
  35.         }
  36.       }
  38.       // 判断如果移除一部分数据之后就可以存放新的block了
  39.       if (actualFreeMemory + selectedMemory >= space) {
  40.         logInfo(s"${selectedBlocks.size} blocks selected for dropping")
  41.         // 将之前选择的要移除的block数据遍历
  42.         for (blockId <- selectedBlocks) {
  43.           val entry = entries.synchronized { entries.get(blockId) }
  44.           // This should never be null as only one thread should be dropping
  45.           // blocks and removing entries. However the check is still here for
  46.           // future safety.
  47.           if (entry != null) {
  48.             val data = if (entry.deserialized) {
  49.               Left(entry.value.asInstanceOf[Array[Any]])
  50.             } else {
  51.               Right(entry.value.asInstanceOf[ByteBuffer].duplicate())
  52.             }
  53.             
  54.             // 调用dropFromMemory方法将尝试数据写入磁盘 , 但是如果block的持久化级别没有说可以写入磁盘那么这个数据就丢咯
  55.             val droppedBlockStatus = blockManager.dropFromMemory(blockId, data)
  56.             droppedBlockStatus.foreach { status => droppedBlocks += ((blockId, status)) }
  57.           }
  58.         }
  59.         return ResultWithDroppedBlocks(success = true, droppedBlocks)
  60.       } else {
  61.         logInfo(s"Will not store $blockIdToAdd as it would require dropping another block " +
  62.           "from the same RDD")
  63.         return ResultWithDroppedBlocks(success = false, droppedBlocks)
  64.       }
  65.     }
  66.     ResultWithDroppedBlocks(success = true, droppedBlocks)
  67.   }
上面在删除一些内存中的老数据时不是真正的删掉 , 而是将数据进行硬盘持久化 , 若是一个block的数据没有设置成硬盘持久化的话那么这份数据就会丢掉 
 而对于硬盘的数据存储就比较简单了 , 源码如下:
  1. override def putBytes(blockId: BlockId, _bytes: ByteBuffer, level: StorageLevel): PutResult = {
  2.     // So that we do not modify the input offsets !
  3.     // duplicate does not copy buffer, so inexpensive
  4.     // 使用Java NIO 将数据写入磁盘文件
  5.     val bytes = _bytes.duplicate()
  6.     logDebug(s"Attempting to put block $blockId")
  7.     val startTime = System.currentTimeMillis
  8.     val file = diskManager.getFile(blockId)
  9.     val channel = new FileOutputStream(file).getChannel
  10.     while (bytes.remaining > 0) {
  11.       channel.write(bytes)
  12.     }
  13.     channel.close()
  14.     val finishTime = System.currentTimeMillis
  15.     logDebug("Block %s stored as %s file on disk in %d ms".format(
  16.       file.getName, Utils.bytesToString(bytes.limit), finishTime - startTime))
  17.     PutResult(bytes.limit(), Right(bytes.duplicate()))
  18.   }

写完数据之后就是报告BlockManagerMasterActor了 , 其实就一行代码:
  1. // 调用reportBlockStatus()方法将新写入的block数据发送到BlockManagerMasterActor以便于进行block元数据的同步和维护
  2.             reportBlockStatus(blockId, putBlockInfo, putBlockStatus)
可以深入一下reportBlockStatus这个方法 :
  1. private def reportBlockStatus(
  2.       blockId: BlockId,
  3.       info: BlockInfo,
  4.       status: BlockStatus,
  5.       droppedMemorySize: Long = 0L): Unit = {
  6.     val needReregister = !tryToReportBlockStatus(blockId, info, status, droppedMemorySize)
  7.     if (needReregister) {
  8.       logInfo(s"Got told to re-register updating block $blockId")
  9.       // Re-registering will report our new block for free.
  10.       asyncReregister()
  11.     }
  12.     logDebug(s"Told master about block $blockId")
  13.   }
  15.   /**
  16.    * Actually send a UpdateBlockInfo message. Returns the master's response,
  17.    * which will be true if the block was successfully recorded and false if
  18.    * the slave needs to re-register.
  19.    */
  20.   private def tryToReportBlockStatus(
  21.       blockId: BlockId,
  22.       info: BlockInfo,
  23.       status: BlockStatus,
  24.       droppedMemorySize: Long = 0L): Boolean = {
  25.     if (info.tellMaster) {
  26.       val storageLevel = status.storageLevel
  27.       val inMemSize = Math.max(status.memSize, droppedMemorySize)
  28.       val inTachyonSize = status.tachyonSize
  29.       val onDiskSize = status.diskSize
  30.       master.updateBlockInfo(
  31.         blockManagerId, blockId, storageLevel, inMemSize, onDiskSize, inTachyonSize)
  32.     } else {
  33.       true
  34.     }
  35.   }
可以看出就是拿取master发送一个更新BlockInfo的消息而已 , 而BlockManagerMasterActor获取到这个消息之后就会调用如下的代码:
  1. /**
  2.    * 更新blockInfo , 即每个BlockManager上的block信息发生变化都会发送updateBlockInfo请求到BlockManagerMaster , 进行BlockInfo的更新
  3.    */
  4.   private def updateBlockInfo(
  5.       blockManagerId: BlockManagerId,
  6.       blockId: BlockId,
  7.       storageLevel: StorageLevel,
  8.       memSize: Long,
  9.       diskSize: Long,
  10.       tachyonSize: Long): Boolean = {
  12.     if (!blockManagerInfo.contains(blockManagerId)) {
  13.       if (blockManagerId.isDriver && !isLocal) {
  14.         // We intentionally do not register the master (except in local mode),
  15.         // so we should not indicate failure.
  16.         return true
  17.       } else {
  18.         return false
  19.       }
  20.     }
  22.     if (blockId == null) {
  23.       blockManagerInfo(blockManagerId).updateLastSeenMs()
  24.       return true
  25.     }
  27.     // 调用BlockManager的BlockManagerInfo的updateBlockInfo()方法更新block信息
  28.     blockManagerInfo(blockManagerId).updateBlockInfo(
  29.       blockId, storageLevel, memSize, diskSize, tachyonSize)
  31.     // 每一个blcok可能会在多个BlockManager上面
  32.     // 如果将StoreageLevel设置成带着 _2 的这种 , 那么就需要将block replicate一份 , 放到其他BlockManager上
  33.     // 而blockLocations map其实是保存了每个blockId对应的BlcokManagerId的set集合
  34.     // 所以 这里会更新blockLocations中的信息 , 因为是用set存储BlockManagerId , 因此自动就去重了
  35.     var locations: mutable.HashSet[BlockManagerId] = null
  36.     if (blockLocations.containsKey(blockId)) {
  37.       locations = blockLocations.get(blockId)
  38.     } else {
  39.       locations = new mutable.HashSet[BlockManagerId]
  40.       blockLocations.put(blockId, locations)
  41.     }
  43.     if (storageLevel.isValid) {
  44.       locations.add(blockManagerId)
  45.     } else {
  46.       locations.remove(blockManagerId)
  47.     }
  49.     // Remove the block from master tracking if it has been removed on all slaves.
  50.     if (locations.size == 0) {
  51.       blockLocations.remove(blockId)
  52.     }
  53.     true
  54.   }

最后第三步 , 需要将设置了备份的block数据传输到别的BlockManager上去存储 :
在duPut操作的方法中最后一步有如下的操作:
  1. // 重要 : 如果我们的持久化级别是定义了_2的这种后缀,说明需要对block进行备份(replica) ,然后传输到其它节点上 
  2.     if (putLevel.replication > 1) {
  3.       data match {
  4.         case ByteBufferValues(bytes) =>
  5.           if (replicationFuture != null) {
  6.             Await.ready(replicationFuture, Duration.Inf)
  7.           }
  8.         case _ =>
  9.           val remoteStartTime = System.currentTimeMillis
  10.           // Serialize the block if not already done
  11.           if (bytesAfterPut == null) {
  12.             if (valuesAfterPut == null) {
  13.               throw new SparkException(
  14.                 "Underlying put returned neither an Iterator nor bytes! This shouldn't happen.")
  15.             }
  16.             bytesAfterPut = dataSerialize(blockId, valuesAfterPut)
  17.           }
  18.           
  19.           // 调用replicate进行复制操作
  20.           replicate(blockId, bytesAfterPut, putLevel)
  21.           logDebug("Put block %s remotely took %s"
  22.             .format(blockId, Utils.getUsedTimeMs(remoteStartTime)))
  23.       }
  24.     }
  26.     BlockManager.dispose(bytesAfterPut)
  28.     if (putLevel.replication > 1) {
  29.       logDebug("Putting block %s with replication took %s"
  30.         .format(blockId, Utils.getUsedTimeMs(startTimeMs)))
  31.     } else {
  32.       logDebug("Putting block %s without replication took %s"
  33.         .format(blockId, Utils.getUsedTimeMs(startTimeMs)))
  34.     }
  36.     updatedBlocks
其中replicate方法就是对数据进行备份:
  1. private def replicate(blockId: BlockId, data: ByteBuffer, level: StorageLevel): Unit = {
  3.     // Get cached list of peers
  4.     peersForReplication ++= getPeers(forceFetch = false)
  5.     def getRandomPeer(): Option[BlockManagerId] = {
  6.       // If replication had failed, then force update the cached list of peers and remove the peers
  7.       // that have been already used
  8.       if (replicationFailed) {
  9.         peersForReplication.clear()
  10.         peersForReplication ++= getPeers(forceFetch = true)
  11.         peersForReplication --= peersReplicatedTo
  12.         peersForReplication --= peersFailedToReplicateTo
  13.       }
  14.       if (!peersForReplication.isEmpty) {
  15.         Some(peersForReplication(random.nextInt(peersForReplication.size)))
  16.       } else {
  17.         None
  18.       }
  19.     }
  20.     while (!done) {
  21.       
  22.       // 随机获取一个其它的BlockManager
  23.       getRandomPeer() match {
  24.         case Some(peer) =>
  25.           try {
  26.             val onePeerStartTime = System.currentTimeMillis
  27.             data.rewind()
  28.             logTrace(s"Trying to replicate $blockId of ${data.limit()} bytes to $peer")
  29.             
  30.             // 使用BlockTransferService将数据异步写入其他的BlockManager上
  31.             blockTransferService.uploadBlockSync(
  32.               peer.host, peer.port, peer.executorId, blockId, new NioManagedBuffer(data), tLevel)
  33.             logTrace(s"Replicated $blockId of ${data.limit()} bytes to $peer in %s ms"
  34.               .format(System.currentTimeMillis - onePeerStartTime))
  35.             peersReplicatedTo += peer
  36.             peersForReplication -= peer
  37.             replicationFailed = false
  38.             if (peersReplicatedTo.size == numPeersToReplicateTo) {
  39.               done = true  // specified number of peers have been replicated to
  40.             }
  41.           } catch {
  42.             case e: Exception =>
  43.               logWarning(s"Failed to replicate $blockId to $peer, failure #$failures", e)
  44.               failures += 1
  45.               replicationFailed = true
  46.               peersFailedToReplicateTo += peer
  47.               if (failures > maxReplicationFailures) { // too many failures in replcating to peers
  48.                 done = true
  49.               }
  50.           }
  51.         case None => // no peer left to replicate to
  52.           done = true
  53.       }
  54.     }
  55.   }
随机获取一个BlockManagerId然后通过BlockTransferService将数据写入到其它的BlockManager上去 , 

Android7.x SurfaceView源码分析
baiiu
09-24 818
更改layoutParams、surfaceView.setVisibility、滑动等基本任何一个对view的操作都会引起updateWindow，所以我们在对SurfaceView操作的时候要加倍小心，避免引起多次updateWindow，从回调上看，就是避免多次触发sufaceChange。通过调整surfaceViewContainer大小、surfaceHolder.setFixedSize，这三种方式均能触发surfaceChange。
Spark-广播变量源码分析
最新发布
lu070828的博客
09-18 909
创建广播变量1、创建广播变量id2、创建TorrentBroadcast3、TorrentBroadcast会自动调用writeBlocks将数据写入到BlockManager（这里会写两份数据，一份是整体的结果用于本Executor使用广播变量时使用，存储级别是MEMORY_AND_DISK，一份是拆分成块用于其他Executor拉取使用，存储级别是MEMORY_AND_DISK_SER）使用广播变量。
块管理器BlockManager
qq_27639777的博客
11-23 568
文章目录简介BlockManager的创建BlockManager的注册Master与Slave间的消息类型BlockManagerMasterEndpoint接收的消息BlockManagerSlaveEndpoint接收的消息BlockManager的存储实现DiskStoreMemoryStore 简介 Spark中的RDD-Cache、Shuffle-output以及broadcast的存...
Spark-BlockManager
weixin_43726774的博客
04-16 3118
简单说明 BlockManager是管理整个Spark运行时数据的读写，包含数据存储本身，在数据存储的基础之上进行数据读写。由于Spark是分布式的，所有BlockManager也是分布式的，BlockManager本身相对而言是一个比较大的模块，Spark中有非常多的模块：调度模块、资源管理模块等等。BlockManager是另外一个非常重要的模块，BlockManager本身源码量非常大。本...
Spark存储体系——块管理器BlockManager
LINBE_blazers的博客
04-25 2675
BlockManager运行在每个节点上（包括Driver和Executor），提供对本地或远端节点上的内存、磁盘及堆外内存中Block的管理。存储体系从狭义上来说指的就是BlockManager，从广义上来说，则包括整个Spark集群中的各个BlockManager、BlockInfoManager、DiskBlockManager、DiskStore、MemoryManager、Memory...
Spark BlockManager源码分析
chongqueluo2709的博客
12-14 288
Spark BlockManager源码分析 更多资源分享 SPARK 源码分析技术分享(视频汇总套装视频): https://www.bilibili.com/video/av37442139/ github: https://github.com/opensourceteams/spar...
spark-BlockManager源码学习
weixin_45448034的博客
06-18 1088
可以看到写代码很简单，不过有两个重点需要留意下，第一个就是内存管理器memoryManager模块，这是一个很重要的模块，其管理我们内存的使用与清理。第二个重点是将数据封装成entry对象存储，entry有两个实现类，分别是SerializedMemoryEntry和DeserializedMemoryEntry，表示序列化和反序列化后的entry信息，从这可以看出，spark内存存储默认都是要序列化的，序列化后会放在LinkedHashMap集合。是的，这里主要实现父类的抽象方法。
Spark BlockManager原理与源码分析
u013174239的博客
05-24 2581
1、BlockManager原理示意图①Driver上的BlockManagerMaster管理各个节点上BlockManager的元数据信息和维护block的状态信息。②每个节点上BlockManager的每个组件：        DiskStore：负责磁盘上的数据读写        MemoryStore: 负责内存中的数据读写        BlockManagerWorker: 负责远程...
Spark源码分析.pdf
04-16
例如，DAGScheduler如何将作业拆分成Stage，TaskScheduler如何将任务分配到Executor，以及如何利用BlockManager和Storage层实现内存管理。此外，Shuffle服务、网络通信库Tachyon和Akka框架也是深入理解Spark的重要...
kubelet之volume manager源码分析
良凯尔的博客
06-06 1251
kubernetes ceph-csi分析目录导航 基于tag v1.17.4 https://github.com/kubernetes/kubernetes/releases/tag/v1.17.4 概述 volume manager存在于kubelet中，主要是管理卷的attach/detach（与AD controller作用相同，通过kubelet启动参数控制哪个组件来做该操作，后续会详细介绍）、mount/umount等操作。 简介 容器的存储挂载分为两大步： （1）attach； （2）mou
Spark自己的分布式存储系统BlockManager全解析
03-29
BlockManager 是 spark 中至关重要的一个组件，在spark的运行过程中到处都有 BlockManager 的身影，只有搞清楚 BlockManager 的原理和机制，你才能更加深入的理解 spark。
BlockManager
何瑞龙的专栏
05-20 770
每次rdd调用iterator方法时，都先会从 final def iterator(split: Partition, context: TaskContext): Iterator[T] = {     if (storageLevel != StorageLevel.NONE) {       SparkEnv.get.cacheManager.getOrCom
BlockManager源码分析
m0_46449152的博客
11-08 307
BlockManagerMasterActor /** * 负责维护各个executor的blockManager的元数据: * blockManagerInfo,BlockStatus */ private[spark] class BlockManagerMasterActor(val isLocal: Boolean, conf: SparkConf, listenerBus: LiveListenerBus) extends Actor with ActorLogReceive with
Spark | BlockManager
俗人
12-04 286
BlockManager是整个Spark存储模块中的核心，它不仅提供存储模块处理各种存储方式的读写方法，而且还为Shuffle模块提供数据处理等操作接口。 BlockManager存在于Driver端和每个Executor中：Driver端的BlockManager 保存了数据的元数据信息；而在Executor中的BlockManager则根据接受到的消息类型进行操作： 当Executor的...
Spark:BlockManager原理剖析与源码分析
标题
01-10 3641
BlockManager是Spark的分布式存储系统，与我们平常说的分布式存储系统是有区别的，区别就是这个分布式存储系统只会管理Block块数据，它运行在所有节点上。BlockManager的结构是Maser-Slave架构，Master就是Driver上的BlockManagerMaster，Slave就是每个Executor上的BlockManager。BlockManagerMaster负责...
Spark源码分析之BlockManager
happy19870612's blog
11-10 1007
BlockManager是对外提供的统一访问block的接口，在Master和Slave上都有一个实例，他提供读写数据的方法，并且根据不同StorageLevel调用不同的BlockStore来读写数据。 在应用程序启动的时候，SparkContext会创建Driver端的SpakEnv，在该SparkEnv中实例化BlockManager和BlockManagerMaster，在其内部创建消息
spark存储体系——BlockManager
weixin_43839095的博客
06-07 982
BlockManager类是在driver和executor上运行，提供了存储的接口。用户只需要调用BlockManager相关的方法就可以完成存储相关的功能。
AutoSar NVM模块详解与源码分析
在该架构中，NVM（Non-Volatile Memory Manager，非易失性存储管理器）作为服务层的一个核心模块，承担着关键的数据持久化与管理职责。本文围绕“AutoSAR NVM介绍[项目源码]”这一主题，深入剖析其架构定位、功能...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值