scrapy将数据存储到mysql中

不凡De老五

于 2018-01-05 10:48:27 发布

阅读量1k

点赞数 1

分类专栏：爬虫

本文链接：https://blog.youkuaiyun.com/weixin_38748717/article/details/78978502

版权

爬虫专栏收录该内容

17 篇文章

订阅专栏

本文详细介绍如何在Scrapy项目中配置MySQL数据库，并通过编写MysqlPipeline实现爬取数据的存储。具体步骤包括：在setting.py文件中设置数据库连接参数；创建MysqlPipeline类并实现数据插入操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一、在配置中设置数据库

#setting.py
ITEM_PIPELINES = {
    ......
   'dushuproject.pipelines.MysqlPipeline': 299,
   }
DB_HOST = '127.0.0.1'
DB_PORT = 3306
DB_USER = 'root'
DB_PWD = '123456'
DB_NAME = 'test'
DB_CHARSET = 'utf8'

二、在管道中进行存储

class MysqlPipeline(object):

    def __init__(self):
        settings = get_project_settings()
        self.host = settings['DB_HOST']
        self.port = settings['DB_PORT']
        self.user = settings['DB_USER']
        self.pwd = settings['DB_PWD']
        self.name = settings['DB_NAME']
        self.charset = settings['DB_CHARSET']

        self.connect()

    def connect(self):
        self.conn = pymysql.connect(host=self.host,
                             port=self.port,
                             user=self.user,
                             password=self.pwd,
                             db=self.name,
                             charset=self.charset)
        self.cursor = self.conn.cursor()

    def close_spider(self, spider):
        self.conn.close()
        self.cursor.close()

    def process_item(self, item, spider):
        sql = 'insert into book(image_url, book_name, author, info) values("%s", "%s", "%s", "%s")' % (item['image_url'], item['book_name'], item['author'], item['info'])
        # 执行sql语句
        self.cursor.execute(sql)
        return item