11、网页抓取的规范与挑战应对

网页抓取的规范与挑战应对

1. 网页抓取规范

1.1 爬虫实现

爬虫以Scrapy蜘蛛的形式实现，以下是一个简单示例：

import scrapy
from scrapy.crawler import CrawlerProcess

class Spider(scrapy.Spider):
    name = 'spider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for link in response.css('div.prev-post > a'):
            yield response.follow(link, self.parse)

    @classmethod
    def close(cls, spider, reason):
        start_time = spider.crawler.stats.get_value('start_time')
        finish_time = spider.crawler.stats.get_value('finish_time')
        print("Total run time: ", finish_time - start_time)

if __name__ == "__main__":
    process = CrawlerProcess({
        'DOWNLOAD_DELAY': 5,
        'RANDOMIZED_DOWNL