11、网页抓取：行为准则与挑战应对

网页抓取：行为准则与挑战应对

1. 爬虫基础实现

爬虫可以使用Scrapy框架实现为一个蜘蛛（Spider）。以下是一个简单的示例代码：

import scrapy
from scrapy.crawler import CrawlerProcess

class Spider(scrapy.Spider):
    name = 'spider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        # 查找CSS选择器 'div.prev-post > a' 并跟随链接
        pass

    @classmethod
    def close(cls, spider, reason):
        start_time = spider.crawler.stats.get_value('start_time')
        finish_time = spider.crawler.stats.get_value('finish_time')
        print("Total run time: ", finish_time - start_time)

if __name__ == "__main__":
    process = CrawlerProcess({
        'DOWNLOAD_DELAY': 5,
        'RANDOMIZED_DOWNLOAD_DELAY': False,
        'LOG_LEVEL': 'DEBUG'