scrapy动态user-agent

最新推荐文章于 2024-04-17 13:54:10 发布

zhdan~

最新推荐文章于 2024-04-17 13:54:10 发布

阅读量1.2k

点赞数

CC 4.0 BY-SA版权

分类专栏： scrapy 文章标签： scrapy python

本文链接：https://blog.youkuaiyun.com/zhangdongan1991/article/details/94601294

scrapy默认的user-agent类似：

User-Agent: b'Scrapy/1.6.0 (+https://scrapy.org)'

一般会自己设置动态user-agent，预防被反爬。步骤：

1、修改setting.py

USER_AGENTS = [
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
    "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
    "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
    "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
    "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
    "Mozilla/