flink网络缓冲池

Flink的网络缓冲池NetworkBufferPool。

构造方法需要两个参数，申请的MemorySegment个数和单个MemorySegment的大小，两者乘积是整个内存缓冲池的总容量。

内部存储结构是一个队列，未分配的MemorySegment存储在队列当中，当被分配时会从内存中取出。

try {
   this.availableMemorySegments = new ArrayBlockingQueue<>(numberOfSegmentsToAllocate);
}
catch (OutOfMemoryError err) {
   throw new OutOfMemoryError("Could not allocate buffer queue of length "
         + numberOfSegmentsToAllocate + " - " + err.getMessage());
}

try {
   for (int i = 0; i < numberOfSegmentsToAllocate; i++) {
      availableMemorySegments.add(MemorySegmentFactory.allocateUnpooledOffHeapMemory(segmentSize, null));
   }
}

以上是其构造方法中的主要部分。

 

NetworkBufferPool提供了createBufferPool()方法可以申请本地缓冲池。

this.numTotalRequiredBuffers += numRequiredBuffers;

// We are good to go, create a new buffer pool and redistribute
// non-fixed size buffers.
LocalBufferPool localBufferPool =
   new LocalBufferPool(this, numRequiredBuffers, maxUsedBuffers, owner);

allBufferPools.add(localBufferPool);

需要提供申请的MemorySegment数量和最大使用MemorySegment数量，这些参数也将作为LocalBufferPool的构造方法参数，之后将其存放在去Set中便于管理。

在建立完毕LocalBufferPool并放入容器当中之后会在createBufferPool()方法中通过redistributeBuffers()方法重新分配网络缓冲池中的内存分配。

final int numAvailableMemorySegment = totalNumberOfMemorySegments - numTotalRequiredBuffers;

if (numAvailableMemorySegment == 0) {
   // in this case, we need to redistribute buffers so that every pool gets its minimum
   for (LocalBufferPool bufferPool : allBufferPools) {
      bufferPool.setNumBuffers(bufferPool.getNumberOfRequiredMemorySegments());
   }
   return;
}

如果当前，网络缓冲池中的内存正巧耗尽 ，那么将LocalBufferPool容器中的LocalBufferPool中的内存数量全部设为最小并返回。

long totalCapacity = 0; // long to avoid int overflow

for (LocalBufferPool bufferPool : allBufferPools) {
   int excessMax = bufferPool.getMaxNumberOfMemorySegments() -
      bufferPool.getNumberOfRequiredMemorySegments();
   totalCapacity += Math.min(numAvailableMemorySegment, excessMax);
}

// no capacity to receive additional buffers?
if (totalCapacity == 0) {
   return; // necessary to avoid div by zero when nothing to re-distribute
}

之后遍历容器，依次计算剩余容量和 单个容器距离最大容量的差的 最小值之和为totalCapacity，为理想情况下每个LocalBufferPool可分配到的内存总量。

final int memorySegmentsToDistribute = MathUtils.checkedDownCast(
      Math.min(numAvailableMemorySegment, totalCapacity));

long totalPartsUsed = 0; // of totalCapacity
int numDistributedMemorySegment = 0;
for (LocalBufferPool bufferPool : allBufferPools) {
   int excessMax = bufferPool.getMaxNumberOfMemorySegments() -
      bufferPool.getNumberOfRequiredMemorySegments();

   // shortcut
   if (excessMax == 0) {
      continue;
   }

   totalPartsUsed += Math.min(numAvailableMemorySegment, excessMax);

   // avoid remaining buffers by looking at the total capacity that should have been
   // re-distributed up until here
   // the downcast will always succeed, because both arguments of the subtraction are in the 'int' domain
   final int mySize = MathUtils.checkedDownCast(
         memorySegmentsToDistribute * totalPartsUsed / totalCapacity - numDistributedMemorySegment);

   numDistributedMemorySegment += mySize;
   bufferPool.setNumBuffers(bufferPool.getNumberOfRequiredMemorySegments() + mySize);
}

之后如上，剩余的容量numAvailableMemorySegment和剩余达到最大容量的差totalCapacity两者中的小者也正是接下来需要分配的内存总量。

接下来的LocalBufferPool可多分配的内存量为单 个LocalBufferPool理想可分配内存 占 总理想分配 的比乘上 总可分配内存 最后减去已分配的内存的结果。