【spdk】SPDK “Reduce“ Block Compression Algorithm

SPDK “Reduce” Block Compression Algorithm {#reduce}

参考

1.社区

#https://www.codenong.com/cs106732883/

#速度与压缩比如何兼得？压缩算法在构建部署中的优化

2.官网

3.源码

Overview

The SPDK “reduce” block compression scheme is based on using SSDs for storing compressed blocks of storage and persistent memory for metadata. This metadata includes mappings of logical blocks requested by a user to the compressed blocks on SSD. The scheme described in this document is generic and not tied to any specific block device framework such as the SPDK block device (bdev) framework. This algorithm will be implemented in a library called “libreduce”. Higher-level software modules can built on top of this library to create and present block devices in a specific block device framework. For SPDK, a bdev_reduce module will serve as a wrapper around the libreduce library, to present the compressed block devices as an SPDK bdev.

【翻译】

关于“reduce”块压缩方案的说明主要涉及以下几点：

1. 使用SSD来存储压缩的存储块，并使用持久内存来存储元数据。元数据包括用户请求的逻辑块到SSD上压缩块的映射。
2. 该方案是通用的，并不依赖于任何特定的块设备框架，如SPDK块设备（bdev）框架。
3. 该算法将在名为“libreduce”的库中实现。
4. 可以在此库上构建更高级别的软件模块，以在特定的块设备框架中创建和呈现块设备。
5. 对于SPDK，bdev_reduce模块将作为libreduce库的包装器，将压缩的块设备呈现为SPDK bdev。

简而言之，该方案允许使用SSD进行数据压缩存储，并使用持久内存来存储元数据。这是一个通用的方案，可以与多种块设备框架结合使用。在SPDK中，bdev_reduce模块将充当libreduce库的接口，以提供压缩的块设备。

This scheme only describes how compressed blocks are stored on an SSD and the metadata for tracking those compressed blocks. It relies on the higher-software module to perform the compression algorithm itself. For SPDK, the bdev_reduce module will utilize the DPDK compressdev framework to perform compression and decompression on behalf of the libreduce library.

【翻译】

这个方案只描述了压缩块如何在SSD上存储以及用于追踪这些压缩块的元数据。它依赖于更高级的软件模块来执行压缩算法本身。对于SPDK，bdev_reduce模块将利用DPDK compressdev框架来代表libreduce库执行压缩和解压缩操作。

(Note that in some cases, blocks of storage may not be compressible, or cannot be compressed enough to realize savings from the compression. In these cases, the data may be stored uncompressed on disk. The phrase “compressed blocks of storage” includes these uncompressed blocks.)

【翻译】

（请注意，在某些情况下，存储块可能无法压缩，或者无法压缩到足够实现压缩节省的程度。在这些情况下，数据可能会以未压缩的形式存储在磁盘上。短语“压缩的存储块”包括这些未压缩的块。）

A compressed block device is a logical entity built on top of a similarly-sized backing storage device. The backing storage device must be thin-provisioned to realize any savings from compression for reasons described later in this document. This algorithm has no direct knowledge of the implementation of the backing storage device, except that it will always use the lowest-numbered blocks available on the backing storage device. This will ensure that when this algorithm is used on a thin-provisioned backing storage device, blocks will not be allocated until they are actually needed.

【翻译】

压缩块设备是在同样大小的底层存储设备之上的逻辑实体。底层存储设备必须进行精简配置，以实现压缩节省，原因将在本文档后面部分说明。此算法不知道底层存储设备的实现方式，除非它将始终使用底层存储设备上可用的最低编号的块。这将确保当此算法在精简配置的底层存储设备上使用时，块将在实际需要时才被分配。

The backing storage device must be sized for the worst case scenario, where no data can be compressed. In this case, the size of the backing storage device would be the same as the compressed block device. Since this algorithm ensures atomicity by never overwriting data in place, some additional backing storage is required to temporarily store data for writes in progress before the associated metadata is updated.

【翻译】

底层存储设备必须按最坏情况场景进行配置，即无法压缩任何数据。在这种情况下，底层存储设备的大小将与压缩块设备的大小相同。由于此算法通过从不原地覆盖数据来确保原子性，因此需要一些额外的底层存储来临时存储正在进行的写操作中的数据，直到相关元数据被更新。

Storage from the backing storage device will be allocated, read, and written to in 4KB units for best NVMe performance