Mata.Gao的博客

1.用户代理池downloader middleware实现随机更换User-Agent.fake-useragent库:up to date simple useragent faker with real world database.github-搜索fake-useragenthttps://fake-useragent.herokuapp.com/browsers/0.1.5以上这个ur...

1.parse.urljoin(base,url)的使用from urllib import parse Request(url=parse.urljoin(response.url, url), callback=self.parse_detail)提取出response.url的主域名与url(/111954/)做url的拼接。若url中有域名，拼接时不会用response.url提取出的主域...

1.同步class MysqlPipeline(object): """ 采用同步的机制写入mysql """ def __init__(self): self.conn = pymysql.connect(host="127.0.0.1", user="root", password="ts123456", db="art_sc

custom_settings = { "COOKIES_ENABLED": False, "DOWNLOAD_DELAY": 1, 'DEFAULT_REQUEST_HEADERS': { 'Accept': 'application/json, text/javascript, */*; q=0.01', ...