《动手学深度学习》手动导入数据集产生错误的解决方法

BasicSR入门教程 BasicSR: 图像、视频超分复原增强开源库的入门使用 阅读详情

动手学深度学习

书本网页版 https://zh.gluon.ai/chapter_preface/preface.html

b站视频讲解 https://space.bilibili.com/209599371?spm_id_from=333.788.b_765f7570696e666f.1

书本源代码、pdf及数据集 链接:https://pan.baidu.com/s/1U53gc7ZIXsF1U23x1g8g4A  提取码:f793

 

在运行书本源代码的时候

需要导入数据集(没有数据集的话第一次会下载数据集

但是 可能出现各种各样的错误导致 我们不能自动下载数据集

(网络问题或者后台路径问题等等

参考资料:https://discuss.gluon.ai/t/topic/642/32

先试着加镜像 

set MXNET_GLUON_REPO=https://apache-mxnet.s3.cn-north-1.amazonaws.com.cn/ jupyter notebook

如果 还是RunTimeError 可以试试我的方法

 

我的解决方法是 使用了同学的数据集以及修改源代码 dataset.py

最后有加 查找SHA1值的方法

1. 四个数据集放在C:\Users\OneLine\.mxnet\datasets\fashion-mnist 文件夹内 

(不用解压)

也可以根据网页提示找到原本的下载路径

2. 对比我原来的dataset.py 和同学的dataset.py 稍微修改一下 (建议直接下拉到末尾

dataset.py的位置在 D:\Anaconda\envs\gluon\Lib\site-packages\mxnet\gluon\data\vision

可能不是D盘 要看自己安装软件的位置 ~\mxnet\gluon\data\vision 找到这一串应该没错

我的 :

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# coding: utf-8
# pylint: disable=
"""Dataset container."""
__all__ = ['MNIST', 'FashionMNIST', 'CIFAR10', 'CIFAR100',
           'ImageRecordDataset', 'ImageFolderDataset']

import os
import gzip
import tarfile
import struct
import warnings
import numpy as np

from .. import dataset
from ...utils import download, check_sha1, _get_repo_file_url
from .... import nd, image, recordio, base


class MNIST(dataset._DownloadedDataset):
    """MNIST handwritten digits dataset from http://yann.lecun.com/exdb/mnist

    Each sample is an image (in 3D NDArray) with shape (28, 28, 1).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/mnist
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'mnist'),
                 train=True, transform=None):
	self._train = train
	self._train_data = ('train-images-idx3-ubyte.gz','0cf37b0d40ed5169c6b3aba31069a9770ac9043d')
        self._train_label = ('train-labels-idx1-ubyte.gz','236021d52f1e40852b06a4c3008d8de8aef1e40b')
        self._test_data = ('t10k-images-idx3-ubyte.gz','626ed6a7c06dd17c0eec72fa3be1740f146a2863')
        self._test_label = ('t10k-labels-idx1-ubyte.gz','17f9ab60e7257a1620f4ad76bbbaf857c3920701')
	self._namespace = 'mnist'
        super(MNIST, self).__init__(root, transform)

    def _get_data(self):
        if self._train:
            data, label = self._train_data, self._train_label
        else:
            data, label = self._test_data, self._test_label

        namespace = 'gluon/dataset/'+self._namespace
        data_file = download(_get_repo_file_url(namespace, data[0]),
                             path=self._root,
                             sha1_hash=data[1])
        label_file = download(_get_repo_file_url(namespace, label[0]),
                              path=self._root,
                              sha1_hash=label[1])

        with gzip.open(label_file, 'rb') as fin:
            struct.unpack(">II", fin.read(8))
            label = np.frombuffer(fin.read(), dtype=np.uint8).astype(np.int32)

        with gzip.open(data_file, 'rb') as fin:
            struct.unpack(">IIII", fin.read(16))
            data = np.frombuffer(fin.read(), dtype=np.uint8)
            data = data.reshape(len(label), 28, 28, 1)

        self._data = nd.array(data, dtype=data.dtype)
        self._label = label


class FashionMNIST(MNIST):
    """A dataset of Zalando's article images consisting of fashion products,
    a drop-in replacement of the original MNIST dataset from
    https://github.com/zalandoresearch/fashion-mnist

    Each sample is an image (in 3D NDArray) with shape (28, 28, 1).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/fashion-mnist'
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'fashion-mnist'),
                 train=True, transform=None):
	self._train = train
	self._train_data = ('train-images-idx3-ubyte.gz','0cf37b0d40ed5169c6b3aba31069a9770ac9043d')
        self._train_label = ('train-labels-idx1-ubyte.gz','236021d52f1e40852b06a4c3008d8de8aef1e40b')
        self._test_data = ('t10k-images-idx3-ubyte.gz','626ed6a7c06dd17c0eec72fa3be1740f146a2863')
        self._test_label = ('t10k-labels-idx1-ubyte.gz','17f9ab60e7257a1620f4ad76bbbaf857c3920701')
        self._namespace = 'fashion-mnist'
        super(MNIST, self).__init__(root, transform) # pylint: disable=bad-super-call


class CIFAR10(dataset._DownloadedDataset):
    """CIFAR10 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html

    Each sample is an image (in 3D NDArray) with shape (32, 32, 3).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/cifar10
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar10'),
                 train=True, transform=None):
        self._train = train
        self._archive_file = ('cifar-10-binary.tar.gz', 'fab780a1e191a7eda0f345501ccd62d20f7ed891')
        self._train_data = [('data_batch_1.bin', 'aadd24acce27caa71bf4b10992e9e7b2d74c2540'),
                            ('data_batch_2.bin', 'c0ba65cce70568cd57b4e03e9ac8d2a5367c1795'),
                            ('data_batch_3.bin', '1dd00a74ab1d17a6e7d73e185b69dbf31242f295'),
                            ('data_batch_4.bin', 'aab85764eb3584312d3c7f65fd2fd016e36a258e'),
                            ('data_batch_5.bin', '26e2849e66a845b7f1e4614ae70f4889ae604628')]
        self._test_data = [('test_batch.bin', '67eb016db431130d61cd03c7ad570b013799c88c')]
        self._namespace = 'cifar10'
        super(CIFAR10, self).__init__(root, transform)

    def _read_batch(self, filename):
        with open(filename, 'rb') as fin:
            data = np.frombuffer(fin.read(), dtype=np.uint8).reshape(-1, 3072+1)

        return data[:, 1:].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1), \
               data[:, 0].astype(np.int32)

    def _get_data(self):
        if any(not os.path.exists(path) or not check_sha1(path, sha1)
               for path, sha1 in ((os.path.join(self._root, name), sha1)
                                  for name, sha1 in self._train_data + self._test_data)):
            namespace = 'gluon/dataset/'+self._namespace
            filename = download(_get_repo_file_url(namespace, self._archive_file[0]),
                                path=self._root,
                                sha1_hash=self._archive_file[1])

            with tarfile.open(filename) as tar:
                tar.extractall(self._root)

        if self._train:
            data_files = self._train_data
        else:
            data_files = self._test_data
        data, label = zip(*(self._read_batch(os.path.join(self._root, name))
                            for name, _ in data_files))
        data = np.concatenate(data)
        label = np.concatenate(label)

        self._data = nd.array(data, dtype=data.dtype)
        self._label = label


class CIFAR100(CIFAR10):
    """CIFAR100 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html

    Each sample is an image (in 3D NDArray) with shape (32, 32, 3).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/cifar100
        Path to temp folder for storing data.
    fine_label : bool, default False
        Whether to load the fine-grained (100 classes) or coarse-grained (20 super-classes) labels.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar100'),
                 fine_label=False, train=True, transform=None):
        self._train = train
        self._archive_file = ('cifar-100-binary.tar.gz', 'a0bb982c76b83111308126cc779a992fa506b90b')
        self._train_data = [('train.bin', 'e207cd2e05b73b1393c74c7f5e7bea451d63e08e')]
        self._test_data = [('test.bin', '8fb6623e830365ff53cf14adec797474f5478006')]
        self._fine_label = fine_label
        self._namespace = 'cifar100'
        super(CIFAR10, self).__init__(root, transform) # pylint: disable=bad-super-call

    def _read_batch(self, filename):
        with open(filename, 'rb') as fin:
            data = np.frombuffer(fin.read(), dtype=np.uint8).reshape(-1, 3072+2)

        return data[:, 2:].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1), \
               data[:, 0+self._fine_label].astype(np.int32)


class ImageRecordDataset(dataset.RecordFileDataset):
    """A dataset wrapping over a RecordIO file containing images.

    Each sample is an image and its corresponding label.

    Parameters
    ----------
    filename : str
        Path to rec file.
    flag : {0, 1}, default 1
        If 0, always convert images to greyscale. \
        If 1, always convert images to colored (RGB).
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, filename, flag=1, transform=None):
        super(ImageRecordDataset, self).__init__(filename)
        self._flag = flag
        self._transform = transform

    def __getitem__(self, idx):
        record = super(ImageRecordDataset, self).__getitem__(idx)
        header, img = recordio.unpack(record)
        if self._transform is not None:
            return self._transform(image.imdecode(img, self._flag), header.label)
        return image.imdecode(img, self._flag), header.label


class ImageFolderDataset(dataset.Dataset):
    """A dataset for loading image files stored in a folder structure.

    like::

        root/car/0001.jpg
        root/car/xxxa.jpg
        root/car/yyyb.jpg
        root/bus/123.jpg
        root/bus/023.jpg
        root/bus/wwww.jpg

    Parameters
    ----------
    root : str
        Path to root directory.
    flag : {0, 1}, default 1
        If 0, always convert loaded images to greyscale (1 channel).
        If 1, always convert loaded images to colored (3 channels).
    transform : callable, default None
        A function that takes data and label and transforms them::

            transform = lambda data, label: (data.astype(np.float32)/255, label)

    Attributes
    ----------
    synsets : list
        List of class names. `synsets[i]` is the name for the integer label `i`
    items : list of tuples
        List of all images in (filename, label) pairs.
    """
    def __init__(self, root, flag=1, transform=None):
        self._root = os.path.expanduser(root)
        self._flag = flag
        self._transform = transform
        self._exts = ['.jpg', '.jpeg', '.png']
        self._list_images(self._root)

    def _list_images(self, root):
        self.synsets = []
        self.items = []

        for folder in sorted(os.listdir(root)):
            path = os.path.join(root, folder)
            if not os.path.isdir(path):
                warnings.warn('Ignoring %s, which is not a directory.'%path, stacklevel=3)
                continue
            label = len(self.synsets)
            self.synsets.append(folder)
            for filename in sorted(os.listdir(path)):
                filename = os.path.join(path, filename)
                ext = os.path.splitext(filename)[1]
                if ext.lower() not in self._exts:
                    warnings.warn('Ignoring %s of type %s. Only support %s'%(
                        filename, ext, ', '.join(self._exts)))
                    continue
                self.items.append((filename, label))

    def __getitem__(self, idx):
        img = image.imread(self.items[idx][0], self._flag)
        label = self.items[idx][1]
        if self._transform is not None:
            return self._transform(img, label)
        return img, label

    def __len__(self):
        return len(self.items)

同学的:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# coding: utf-8
# pylint: disable=
"""Dataset container."""
__all__ = ['MNIST', 'FashionMNIST', 'CIFAR10', 'CIFAR100',
           'ImageRecordDataset', 'ImageFolderDataset']

import os
import gzip
import tarfile
import struct
import warnings
import numpy as np

from .. import dataset
from ...utils import download, check_sha1, _get_repo_file_url
from .... import nd, image, recordio, base
from .... import numpy as _mx_np  # pylint: disable=reimported
from ....util import is_np_array


class MNIST(dataset._DownloadedDataset):
    """MNIST handwritten digits dataset from http://yann.lecun.com/exdb/mnist

    Each sample is an image (in 3D NDArray) with shape (28, 28, 1).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/mnist
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'mnist'),
                 train=True, transform=None):
        self._train = train
        self._train_data = ('train-images-idx3-ubyte.gz',
                            '6c95f4b05d2bf285e1bfb0e7960c31bd3b3f8a7d')
        self._train_label = ('train-labels-idx1-ubyte.gz',
                             '2a80914081dc54586dbdf242f9805a6b8d2a15fc')
        self._test_data = ('t10k-images-idx3-ubyte.gz',
                           'c3a25af1f52dad7f726cce8cacb138654b760d48')
        self._test_label = ('t10k-labels-idx1-ubyte.gz',
                            '763e7fa3757d93b0cdec073cef058b2004252c17')
        self._namespace = 'mnist'
        super(MNIST, self).__init__(root, transform)

    def _get_data(self):
        if self._train:
            data, label = self._train_data, self._train_label
        else:
            data, label = self._test_data, self._test_label

        namespace = 'gluon/dataset/'+self._namespace
        data_file = download(_get_repo_file_url(namespace, data[0]),
                             path=self._root,
                             sha1_hash=data[1])
        label_file = download(_get_repo_file_url(namespace, label[0]),
                              path=self._root,
                              sha1_hash=label[1])

        with gzip.open(label_file, 'rb') as fin:
            struct.unpack(">II", fin.read(8))
            label = np.frombuffer(fin.read(), dtype=np.uint8).astype(np.int32)
            if is_np_array():
                label = _mx_np.array(label, dtype=label.dtype)

        with gzip.open(data_file, 'rb') as fin:
            struct.unpack(">IIII", fin.read(16))
            data = np.frombuffer(fin.read(), dtype=np.uint8)
            data = data.reshape(len(label), 28, 28, 1)

        array_fn = _mx_np.array if is_np_array() else nd.array
        self._data = array_fn(data, dtype=data.dtype)
        self._label = label


class FashionMNIST(MNIST):
    """A dataset of Zalando's article images consisting of fashion products,
    a drop-in replacement of the original MNIST dataset from
    https://github.com/zalandoresearch/fashion-mnist

    Each sample is an image (in 3D NDArray) with shape (28, 28, 1).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/fashion-mnist'
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'fashion-mnist'),
                 train=True, transform=None):
        self._train = train
        self._train_data = ('train-images-idx3-ubyte.gz',
                            '0cf37b0d40ed5169c6b3aba31069a9770ac9043d')
        self._train_label = ('train-labels-idx1-ubyte.gz',
                             '236021d52f1e40852b06a4c3008d8de8aef1e40b')
        self._test_data = ('t10k-images-idx3-ubyte.gz',
                           '626ed6a7c06dd17c0eec72fa3be1740f146a2863')
        self._test_label = ('t10k-labels-idx1-ubyte.gz',
                            '17f9ab60e7257a1620f4ad76bbbaf857c3920701')
        self._namespace = 'fashion-mnist'
        super(MNIST, self).__init__(root, transform) # pylint: disable=bad-super-call


class CIFAR10(dataset._DownloadedDataset):
    """CIFAR10 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html

    Each sample is an image (in 3D NDArray) with shape (32, 32, 3).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/cifar10
        Path to temp folder for storing data.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar10'),
                 train=True, transform=None):
        self._train = train
        self._archive_file = ('cifar-10-binary.tar.gz', 'fab780a1e191a7eda0f345501ccd62d20f7ed891')
        self._train_data = [('data_batch_1.bin', 'aadd24acce27caa71bf4b10992e9e7b2d74c2540'),
                            ('data_batch_2.bin', 'c0ba65cce70568cd57b4e03e9ac8d2a5367c1795'),
                            ('data_batch_3.bin', '1dd00a74ab1d17a6e7d73e185b69dbf31242f295'),
                            ('data_batch_4.bin', 'aab85764eb3584312d3c7f65fd2fd016e36a258e'),
                            ('data_batch_5.bin', '26e2849e66a845b7f1e4614ae70f4889ae604628')]
        self._test_data = [('test_batch.bin', '67eb016db431130d61cd03c7ad570b013799c88c')]
        self._namespace = 'cifar10'
        super(CIFAR10, self).__init__(root, transform)

    def _read_batch(self, filename):
        with open(filename, 'rb') as fin:
            data = np.frombuffer(fin.read(), dtype=np.uint8).reshape(-1, 3072+1)

        return data[:, 1:].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1), \
               data[:, 0].astype(np.int32)

    def _get_data(self):
        if any(not os.path.exists(path) or not check_sha1(path, sha1)
               for path, sha1 in ((os.path.join(self._root, name), sha1)
                                  for name, sha1 in self._train_data + self._test_data)):
            namespace = 'gluon/dataset/'+self._namespace
            filename = download(_get_repo_file_url(namespace, self._archive_file[0]),
                                path=self._root,
                                sha1_hash=self._archive_file[1])

            with tarfile.open(filename) as tar:
                tar.extractall(self._root)

        if self._train:
            data_files = self._train_data
        else:
            data_files = self._test_data
        data, label = zip(*(self._read_batch(os.path.join(self._root, name))
                            for name, _ in data_files))
        data = np.concatenate(data)
        label = np.concatenate(label)

        array_fn = _mx_np.array if is_np_array() else nd.array
        self._data = array_fn(data, dtype=data.dtype)
        self._label = array_fn(label, dtype=label.dtype) if is_np_array() else label


class CIFAR100(CIFAR10):
    """CIFAR100 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html

    Each sample is an image (in 3D NDArray) with shape (32, 32, 3).

    Parameters
    ----------
    root : str, default $MXNET_HOME/datasets/cifar100
        Path to temp folder for storing data.
    fine_label : bool, default False
        Whether to load the fine-grained (100 classes) or coarse-grained (20 super-classes) labels.
    train : bool, default True
        Whether to load the training or testing set.
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar100'),
                 fine_label=False, train=True, transform=None):
        self._train = train
        self._archive_file = ('cifar-100-binary.tar.gz', 'a0bb982c76b83111308126cc779a992fa506b90b')
        self._train_data = [('train.bin', 'e207cd2e05b73b1393c74c7f5e7bea451d63e08e')]
        self._test_data = [('test.bin', '8fb6623e830365ff53cf14adec797474f5478006')]
        self._fine_label = fine_label
        self._namespace = 'cifar100'
        super(CIFAR10, self).__init__(root, transform) # pylint: disable=bad-super-call

    def _read_batch(self, filename):
        with open(filename, 'rb') as fin:
            data = np.frombuffer(fin.read(), dtype=np.uint8).reshape(-1, 3072+2)

        return data[:, 2:].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1), \
               data[:, 0+self._fine_label].astype(np.int32)


class ImageRecordDataset(dataset.RecordFileDataset):
    """A dataset wrapping over a RecordIO file containing images.

    Each sample is an image and its corresponding label.

    Parameters
    ----------
    filename : str
        Path to rec file.
    flag : {0, 1}, default 1
        If 0, always convert images to greyscale. \
        If 1, always convert images to colored (RGB).
    transform : function, default None
        A user defined callback that transforms each sample. For example::

            transform=lambda data, label: (data.astype(np.float32)/255, label)

    """
    def __init__(self, filename, flag=1, transform=None):
        super(ImageRecordDataset, self).__init__(filename)
        self._flag = flag
        self._transform = transform

    def __getitem__(self, idx):
        record = super(ImageRecordDataset, self).__getitem__(idx)
        header, img = recordio.unpack(record)
        if self._transform is not None:
            return self._transform(image.imdecode(img, self._flag), header.label)
        return image.imdecode(img, self._flag), header.label


class ImageFolderDataset(dataset.Dataset):
    """A dataset for loading image files stored in a folder structure.

    like::

        root/car/0001.jpg
        root/car/xxxa.jpg
        root/car/yyyb.jpg
        root/bus/123.jpg
        root/bus/023.jpg
        root/bus/wwww.jpg

    Parameters
    ----------
    root : str
        Path to root directory.
    flag : {0, 1}, default 1
        If 0, always convert loaded images to greyscale (1 channel).
        If 1, always convert loaded images to colored (3 channels).
    transform : callable, default None
        A function that takes data and label and transforms them::

            transform = lambda data, label: (data.astype(np.float32)/255, label)

    Attributes
    ----------
    synsets : list
        List of class names. `synsets[i]` is the name for the integer label `i`
    items : list of tuples
        List of all images in (filename, label) pairs.
    """
    def __init__(self, root, flag=1, transform=None):
        self._root = os.path.expanduser(root)
        self._flag = flag
        self._transform = transform
        self._exts = ['.jpg', '.jpeg', '.png']
        self._list_images(self._root)

    def _list_images(self, root):
        self.synsets = []
        self.items = []

        for folder in sorted(os.listdir(root)):
            path = os.path.join(root, folder)
            if not os.path.isdir(path):
                warnings.warn('Ignoring %s, which is not a directory.'%path, stacklevel=3)
                continue
            label = len(self.synsets)
            self.synsets.append(folder)
            for filename in sorted(os.listdir(path)):
                filename = os.path.join(path, filename)
                ext = os.path.splitext(filename)[1]
                if ext.lower() not in self._exts:
                    warnings.warn('Ignoring %s of type %s. Only support %s'%(
                        filename, ext, ', '.join(self._exts)))
                    continue
                self.items.append((filename, label))

    def __getitem__(self, idx):
        img = image.imread(self.items[idx][0], self._flag)
        label = self.items[idx][1]
        if self._transform is not None:
            return self._transform(img, label)
        return img, label

    def __len__(self):
        return len(self.items)

主要修改了两个地方

(因为数据集和源代码都来自同学 所以SHA值不需要修改了)

(1)删除红框里面的内容 (因为在另一个py文件里没有包含对numpy包的导入

(2) 删除与numpy 和 is_np_array有关的函数

 

1的部分全部删除 

2的部分改成

        self._data = nd.array(data, dtype=data.dtype)
        self._label = label

后面的部分函数也有包含is_np_array 但是我没删除 (因为没有影响我做的实验 如果有影响的话再删除8

 

最后写一下 我自己查找SHA1值的方法 虽然后来没改变SHA1值

在对应的文件夹目录栏输入 cmd 跳转之后输入certutil -hashfile 文件名 SHA1

大概这样子啦~~

Accelerate分布式训练实战:零修改迁移单卡PyTorch模型 分布式训练是深度学习工程化的核心能力,其本质是通过多设备协同提升模型训练效率与规模上限。原理上依赖数据并行、梯度同步与混合精度等关键技术,技术价值在于降低显存压力、加速收敛、支撑大模型落地。典型应用场景包括多卡GPU训练、跨机集群微调、生产环境CI/CD集成及Hugging Face生态适配。本文聚焦PyTorch原生分布式封装工具Accelerate,围绕‘零代码侵入’和‘自动设备管理’两大热词,详解如何不改训练循环即可实现单卡→多卡→多机的平滑演进,覆盖初始化配置、prepare机制、backward替 阅读详情

相关推荐

跨学科AI学习路线:从问题出发,快速跑通论文与项目闭环

跨学科研究者接触人工智能时,常被从底层原理开始的学习路径吓退。事实上,大语言模型(LLM)并非遥不可及,其核心是基于海量文本的统计规律生成内容,而检索增强生成(RAG)则通过外部知识检索显著减少幻觉,让模型输出可溯源。Prompt设计作为与模型高效交互的关键,能精准约束任务边界,配合Agent的自动化能力,可在文献综述、选题验证、实验设计等科研场景中大幅提升效率。从概念原理到工程实践,合理运用这些技术,能帮助非计算机背景的学者快速构建“问题→数据→方法→验证”的最小闭环。本文提供一套以论文和项目为目标的四阶

weixin_33984032的博客 494

如何解决成功安装了第三方库,import导入时却没有?

*第三方模块不能在虚拟环境中使用,第三方模块不能在虚拟环境中使用,第三方模块不能在虚拟环境中使用。**重要的是只说三遍!绝大多数第三方库安装成功无法使用都是这个问题,明白这个就很容易了。 接下来以图片形式教大家如何修改自己电脑的环境。 点击左上角file 点击new projict…创建一个新项目 勾选previously configured interlreter框 点击右边的3个小点 选择System Interpreter 选择好Interpreter地址创建项目即可 ...

m0_62984290的博客 1756

BasicSR的使用过程

由于最近在研究BasicSR的超分辨率的各种模型,在测试各种实验的过程中出现过很多的问题,下面针对这些问题说一下解决方式。比如像我本次遇到的问题,当输入的图像数据特别小时我们应该如何处理。本文仅针对使用BasicSR进行一个模型使用和模型对比的安装、使用、一些注意实现进行一部分的描述。

qq_38735174的博客 8234

【报错】ImportError: cannot import name ‘get_refined_artifact_map‘ from ‘basicsr.losses.loss_util‘

缺失的部分如下,补充到原来的xxx/lib/python3.10/site-packages/basicsr/losses/loss_util.py下面。

Ppandaer的博客 395

9-Dataset创建-import调用

【代码】9-2-Dataset创建-import调用。

MaYuHuaw的博客 1394

中华白海豚个体识别:深度学习ReID全流程落地指南

目标检测与特征提取是计算机视觉的基础任务,当识别对象从通用物体转向野生动物个体时,算法需要面对开放集分类与跨时空匹配的挑战。基于深度学习的个体识别(ReID)通过检测模型框出目标局部特征,再利用三元组损失训练特征向量,使同一只个体的不同照片在特征空间中彼此靠近,不同个体相互远离。该技术已从行人重识别延伸到中华白海豚等海洋生物保护领域,解决了背鳍轮廓、疤痕色斑等细微差异带来的长期监测难题。在实际工程中,需依次完成数据标注、骨干网络选型、PK采样、数据增强与置信度分桶,并配合人工复核形成闭环,才能将秒级匹配结果

weixin_33754065的博客 466

动手学深度学习下载数据集慢且出现RuntimeError:Failed downloading url...错误

 今天做深度学习的实验时,需要用到Fashion-MNIST数据集,在运行书本源代码下载数据集的时候,下载很慢,直到出现RuntimeError错误。  首先试着通过指定MXNet使⽤国内站点下载书中的数据和模型: set MXNET_GLUON_REPO=https://apache-mxnet.s3.cn-north-1.amazonaws.com.cn/ jupyter notebook 接着,出现如下错误:  查阅资料,找到解决办法:通过修改C:\Users\27772\Miniconda\en

芳雅葳蕤的博客 2425

BasicSR:用于超分辨率,去噪,去模糊等的开源图像和视频恢复工具箱。当前,它包括EDSR,RCAN,SRResNet,SRGAN,ESRGAN,EDVR等。还支持StyleGAN2,DFDNet

:rocket: 基本SR | | Google Colab: | :circled_M: :fast_down_button: Google云端硬盘: | :fast_down_button:百度网盘:| :file_folder: :fast_down_button: :fast_down_button:(提取码:basr) :chart_increasing: :laptop: :high_voltage: BasicSR(基本S- UPERřestoration)是基于PyTorch一个开源图像和视频恢复工具箱,如超分辨率,降噪,去模糊,JPEG伪像的去除,等等。 ( , , , ) ( , , , ) :sparkles: 新的功能 2020年11月29日。添加ESRGAN和DFDNet 。 2020年9月8日。添加盲人脸恢复推理代码: 。 2020年8月27日。添加StyleGAN2培训和测试代码: 。 更多的 2020年9月8日。添加盲人脸恢复推理代码: DFDNet 。 ECCV20:通过深度多尺度组件字典进行盲人脸恢

【李沐 | 动手学深度学习】 22 Classify-Leaves 任务实现

本文介绍了一个基于ResNet50的叶子图像分类竞赛解决方案。任务目标是预测176类叶子图像,数据集包含18,353张训练图像和8,800张测试图像。采用迁移学习策略,使用ImageNet预训练的ResNet50模型作为基础模型。技术实现包括:1)定义LeavesDataset类处理数据;2)采用7:3比例划分训练/验证集,并进行数据增强;3)修改模型最后一层输出维度;4)使用交叉熵损失和标签平滑技术。实验结果显示模型验证准确率达到90%以上,经过二次微调后提升至94%。文章还讨论了5折交叉验证的潜在应用,

m0_74045201的博客 399

Outlook邮件备份全攻略:从导出到导入的完整流程(附常见问题解决)

本文提供了一份详尽的Outlook邮件备份全攻略,涵盖从导出到导入的完整流程。重点解析了经典桌面版Outlook的.pst文件备份方法、进阶的智能筛选备份策略,以及如何通过导入恢复和工具验证确保备份可用。同时,文章还介绍了构建自动化备份体系与异地容灾的最佳实践,并附有常见问题解决方案,帮助用户建立可靠的邮件数据安全防线。

weixin_29248611的博客 891

从URDF到仿真动画:基于Simulink、Simscape与S-Function的机械臂一体化建模与控制

本文详细介绍了从URDF文件到高保真仿真动画的一体化机械臂建模与控制流程。通过整合Simulink、Simscape与S-Function,构建了包含物理特性的虚拟机械臂模型,并实现了自适应滑模等高级控制算法的嵌入与验证。该流程打通了从机械设计到控制算法验证的闭环,为机器人开发提供了高效的仿真平台。

cnn8visionary的博客 697

从零到一:AI学习路径与PyTorch实战手写数字识别

人工智能作为当前技术发展的核心驱动力,其底层原理基于机器学习与深度学习算法。通过理解神经网络、梯度下降等基础概念,开发者可以掌握模型构建与训练的核心机制。在工程实践中,PyTorch等框架极大地降低了AI应用的开发门槛,其动态计算图特性便于快速原型验证。结合具体项目,如手写数字识别,能够将理论知识转化为解决实际问题的能力,这正是AI技术价值的体现。随着AI工具链的演进,例如Wayfinder Router这类工具的出现,AI应用正朝着更易集成、更易部署的方向发展,为开发者提供了从模型训练到生产落地的完整解决

weixin_33701564的博客 409

Ubuntu22.04安装搜狗输入法踩坑实录:从依赖报错到完美解决的完整指南

本文详细解析了在Ubuntu 22.04系统上安装搜狗拼音输入法的完整流程,从解决依赖报错到配置fcitx框架,再到最终优化使用体验。针对常见的dpkg安装错误和fcitx冲突问题,提供了实用的解决方案和命令,帮助用户顺利完成安装并享受流畅的中文输入体验。

palm99的专栏 971

NLP数据基础设施:标准化接口、可复现预处理与版本化元信息

在NLP工程实践中,‘数据可用性’远比‘数据存在性’更关键——它涉及文本、标签、序列结构等多维信息的强类型约束、确定性清洗与跨环境一致性保障。其核心原理在于通过Schema定义数据契约、Transform实现幂等清洗、Loader完成可验证加载,从而构建具备工业级可靠性的数据流。该架构显著降低因编码差异、隐式依赖或字段错配引发的线上故障,支撑金融、医疗、社交等多场景高稳定性NLP产线落地。本文聚焦NLP Dataset Library如何以标准化接口、可复现预处理和版本化元信息三大能力,解决真实业务中的数据

wyjvip333的专栏 178

第1.1节 构网型 vs. 跟网型:根本区别与技术原理

构网型与跟网型变流器的本质区别在于控制哲学与等效模型。跟网型变流器采用"电压跟随,电流注入"策略,等效为受控电流源,依赖电网电压作为参考,在弱电网下存在稳定性问题。构网型变流器采用"自主同步,电压构建"策略,等效为受控电压源,能主动构建电压和频率,具备自主运行、黑启动和电网支撑能力。两者的核心差异体现在同步机制、电网依赖性、故障行为和系统支撑等方面。构网型变流器更适合高比例新能源、弱电网和微电网场景,是未来电力系统的重要发展方向。

技术工作者 1394

java 二维码生成及其标签打印

本文主要内容 二维码生成 二维码标签预览及打印 二维码生成 笔者此次的二维码是通过调用第三方接口生成的,具体流程如下: 根据规范要求调用第三方接口,返回二维码下载地址及二维码图片的属性值(图片大小等) 根据返回的值获取二维码下载地址,根据地址下载二维码图片,保存至本地磁盘及记录数据信息 扫描二维码跳转相应地址,显示资产的基本信息 最终生成的二维码如下图所示: 预览并批量打印二维码标签 先...

孙小牟的博客 4941

import dgl 图神经网络库的导入报错问题

dgl库的导入

weixin_45695606的博客 2238

常用的串口通信工具MobaXterm、Bus Hound、SScom等

本文介绍了四种PC端串口通信工具的功能和使用方法:MobaXterm支持SSH、FTP、串口等多种协议;Bus Hound用于监控CAN和串口数据;SScom专为串口和网口通信设计;ScriptCommunicator可进行串口通信并绘制波形图。每种工具都有详细的操作步骤图示,包括参数设置、数据监控等关键功能,并提供了官方和个人下载地址。这些工具各有特点,适用于不同的串口通信和数据分析场景。

sanqima的专栏 2485

GAN 中Loss函数精讲

GANs有生成器Generator和辨别器Discriminator组成,基本的结构图如下: GANs结构示意图 工作原理   我们通常使用两个优化算法来训练GANs。判别器是一个普通的神经网络分类器,训练的过程中,我们使用辨别器 (discriminator) 学习引导生成器。   判别器:   在训练的过程中,我们向辨别器discriminator输入的数据一...

gxiaoyaya的博客 2万+

Comsol超声空化气泡仿真:从基础理论到工业应用实践

本文详细介绍了Comsol超声空化气泡仿真的基础理论及工业应用实践。通过解析超声空化现象、Comsol模型搭建流程及求解器优化技巧,帮助工程师掌握气泡仿真技术,提升超声波清洗和乳液制备等工业应用的效率与质量。

weixin_27966471的博客 251

车载ECU底软测试:方法与技术的深度解析

底软是运行在 ECU 硬件平台上的基础软件,它为上层应用程序提供了硬件驱动、操作系统服务、通信协议栈等功能。底软的主要功能包括:硬件驱动:负责与 ECU 的各种硬件外设进行通信和控制,如 GPIO、UART、SPI、CAN 等。操作系统服务:提供任务调度、内存管理、中断处理等功能,确保系统的实时性和稳定性。通信协议栈:实现 ECU 与其他 ECU 或外部设备之间的通信,如 CAN、LIN、FlexRay 等。诊断服务:负责对 ECU 的故障进行诊断和处理,并将故障信息存储在故障码存储器中。

天赐好车的博客 1425

datasets Dataset类(1)

函数说明 根据ArrowTable产生一个Dataset对象。 根据字典来创建Dataset数据集对象。 2、from_pandas函数 根据pandas中的DataFrame对象创建Dataset对象。 3、from_csv函数 根据csv文件来创建Dataset对象。 根据json文件来创建Dataset对象。 根据txt文件来创建Dataset对象。 根据parquet文件来创建Dataset对象。 根据arrow文件来创建Dataset对象。 根据arrow buffer对象来创建Dataset

weixin_49346755的博客 5133

TMS_WebGMaps_1.9.8.0_Full_Source

TMS_WebGMaps_1.9.8.0_Full_Source 全源码 delphi cbuilder

Gan的loss函数

1. Classic Adversarial Loss 优化目标为: D(x)为经过sigmoid的输出值 (1)在GAN第一阶段——求Discriminator,最大化 实验中统计梯度是对最小值进行寻优的,因此实际操作上是对目标函数做最小化处理: 实现方式: dis_real、dis_fake分别为真假样本输入判别器输出的logit,未经过sigmoid (1)直接计算 L1 = F.mean(-F.log(F.sigmoid(dis_real))...

qq_38101208的博客 3001

温度PID控制PLC程序

温度PID稳定控制程序,适用于欧姆龙PLC,对其他PLC也具有很大参考价值

STM32F407中文手册

STM32f407中文手册 本参考手册面向应用开发人员,提供有关使用 STM32F405xx/07xx、STM32F415xx/17xx、 STM32F42xxx 和 STM32F43xxx 微控制器存储器与外设的完整信息。STM32F405xx/07xx、STM32F415xx/17xx、STM32F42xxx 和 STM32F43xxx 构成一个微 控制器系列,各产品具有不同的存储器大小、封装和外设。有关订购信息以及器件的机械与电气特性,请参见数据手册。有关 ARM Cortex™-M4F 内核的信息,请参见《Cortex™-M4F 技术参考手册》。

Python · House Prices - Advanced Regression TechniquesKaggle实战:预测房价

1.动手学深度学习课程版本

pxyp123的博客 1586

WINCC7.3破解文件

WINCC7.3硬件够破解文件,有需要的下载!WINCC7.3硬件够破解文件,有需要的下载!

【Python】os.path.expanduser() 函数的理解和举例说明

【Python】os.path.expanduser() 函数的理解和举例说明

wzk4869的博客 1万+
上一篇: 虚拟机RedHat利用镜像文件安装gcc、bind等软件包
下一篇: 申请Matlab账号的两种方式 | 学校认证or数模比赛申请 (安利MATLAB online)
OneLine_
博客等级 码龄9年 48粉丝 72原创
评论 2
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值