mxnet进阶 - mxnet.io.DataDesc 源码分析

最新推荐文章于 2024-04-29 15:11:47 发布

原创最新推荐文章于 2024-04-29 15:11:47 发布 · 645 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#mxnet

MXnet 专栏收录该内容

8 篇文章

订阅专栏

本文详细介绍了MXNet中DataDesc类的功能与用法。DataDesc是用于存储数据名称、形状、类型和布局信息的数据描述子，它帮助描述数据集的结构特征，如批处理轴、数据类型等，对于数据并行处理至关重要。

全称应该叫Data Descriptor 数据描述子

一个namedtuple的子类

本质也是一个tuple，只不过对不同位置的元素命名了而已，可以通过命名访问到元素，和ordinaried字典类似

mxnet.DataDesc 使用了两个信息描述数据

name 是名字字符串 str
shape 是数据形状元组里面存int类型的数

class DataDesc(namedtuple('DataDesc', ['name', 'shape'])):
    """DataDesc is used to store name, shape, type and layout
    information of the data or the label.

    The `layout` describes how the axes in `shape` should be interpreted,
    for example for image data setting `layout=NCHW` indicates
    that the first axis is number of examples in the batch(N),
    C is number of channels, H is the height and W is the width of the image.

    For sequential data, by default `layout` is set to ``NTC``, where
    N is number of examples in the batch, T the temporal axis representing time
    and C is the number of channels.

    Parameters
    ----------
    cls : DataDesc
         The class.
    name : str
         Data name.
    shape : tuple of int
         Data shape.
    dtype : np.dtype, optional
         Data type.
    layout : str, optional
         Data layout.
    """
    def __new__(cls, name, shape, dtype=mx_real_t, layout='NCHW'): # pylint: disable=super-on-old-class
        ret = super(cls, DataDesc).__new__(cls, name, shape)
        ret.dtype = dtype
        ret.layout = layout
        return ret

    def __repr__(self):
        return "DataDesc[%s,%s,%s,%s]" % (self.name, self.shape, self.dtype,
                                          self.layout)

    @staticmethod
    def get_batch_axis(layout):
        """Get the dimension that corresponds to the batch size.

        When data parallelism is used, the data will be automatically split and
        concatenated along the batch-size dimension. Axis can be -1, which means
        the whole array will be copied for each data-parallelism device.

        Parameters
        ----------
        layout : str
            layout string. For example, "NCHW".

        Returns
        -------
        int
            An axis indicating the batch_size dimension.
        """
        if layout is None:
            return 0
        return layout.find('N')

    @staticmethod
    def get_list(shapes, types):
        """Get DataDesc list from attribute lists.

        Parameters
        ----------
        shapes : a tuple of (name, shape)
        types : a tuple of  (name, type)
        """
        if types is not None:
            type_dict = dict(types)
            return [DataDesc(x[0], x[1], type_dict[x[0]]) for x in shapes]
        else:
            return [DataDesc(x[0], x[1]) for x in shapes]