scrapy 下载gif图片

最新推荐文章于 2022-07-19 12:00:00 发布

转载最新推荐文章于 2022-07-19 12:00:00 发布 · 1.4k 阅读

scrapy 专栏收录该内容

5 篇文章

订阅专栏

本文介绍如何使用scrapy的imagespipeline自定义下载特定格式的GIF图像，通过重写image_downloaded方法来实现直接保存GIF文件。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scrapy 的images pipeline 默认将图片转换成通用的格式（JPG）和模式（RGB）

Come across to this thread. Here's my MyImagesPipeline to download full size gif images by overriding image_downloaded.

class MyImagesPipeline(ImagesPipeline):

    def check_gif(self, image):
        if image.format == 'GIF':
            return True
        # The library reads GIF87a and GIF89a versions of the GIF file format.
        return image.info.get('version') in ['GIF89a', 'GIF87a']

    def persist_gif(self, key, data, info):
        root, ext = os.path.splitext(key)
        key = key + '.gif'
        absolute_path = self.store._get_filesystem_path(key)
        self.store._mkdir(os.path.dirname(absolute_path), info)
        f = open(absolute_path, 'wb')   # use 'b' to write binary data.
        f.write(data)

    def image_downloaded(self, response, request, info):
        checksum = None
        for key, image, buf in self.get_images(response, request, info):
            if checksum is None:
                buf.seek(0)
                checksum = md5sum(buf)
            if key.startswith('full') and self.check_gif(image):
                # Save gif from response directly.
                self.persist_gif(key, response.body, info)
            else:
                self.store.persist_image(key, image, buf, info)
        return checksum