- 博客(9)
- 资源 (1)
- 收藏
- 关注
RSS订阅原创 爬取汽车之家
import scrapyfrom scrapy.linkextractors import LinkExtractorfrom scrapy.spiders import CrawlSpider, Rulefrom BWM3.items import Bwm3Itemclass Bwm3GtSpider(CrawlSpider):name = ‘bwm3_gt’allowed_domains = [‘car.autohome.com.cn’]start_urls = [‘https://ca
2020-07-20 12:16:11
235
原创 爬取ajax斗鱼
import scrapyimport jsonfrom douyu.items import DouyuItemclass DySpider(scrapy.Spider):name = ‘dy’allowed_domains = [‘douyu.com’]base_url = ‘http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offset=’# 对应多少间隔offsert = 0# 爬取的起始网址start_url
2020-07-19 14:44:51
162
原创 start代码
from scrapy import cmdlinecmdline.execute([“scrapy”,“crawl”,‘爬虫名’])
2020-07-19 14:43:59
436
原创 scrapy爬取80小说
-- coding: utf-8 --import scrapyfrom Novels1.items import Novels1Itemclass Novels80Spider(scrapy.Spider):name = ‘novels80’allowed_domains = [‘txt80.com’]start_urls = [‘http://txt80.com/dushi/’]def parse(self, response): # divs = response.xpath('
2020-07-19 14:43:14
205
原创 scrapy爬取当当
import scrapy上一个文件夹的item的DangdangItemfrom …items import DangdangItemclass DdSpider(scrapy.Spider):name = ‘dd’allowed_domains = [‘dangdang.com’]start_urls = [‘http://search.dangdang.com/?key=python’]def parse(self, response): # 使用xpath获取包含所有书籍信息的
2020-07-19 14:40:04
169
原创 爬取广西防疫信息
import requestsimport timefrom lxml import etreeimport re9 12 5 7headers = {‘User-Agent’ : ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36’}获取到五月份数据的urldef get_May_info(url):resp
2020-07-19 14:37:26
183
原创 python连接mongo
from pymongo import MongoClientclass A:def init(self):# 连接mongodbself.client = MongoClient()# 确定连接的数据库self.db = self.client[‘application’]def addOne(self): obj = {'name':'特斯拉'} return self.db.car.insert_one(obj)def addMany(self): obj =
2020-07-19 14:36:37
184
原创 xpath练习
‘’’2020.5.28作业xpath写法‘’’import requestsfrom lxml import etreeimport socketimport timeimport randomfrom pymongo import MongoClient白嫖的代理ipproxies = {‘https’ : ‘58.218.201.74:2874’}设置代理def getUserAgent():“”":return: 随机返回一个浏览器请求头的User-Agent值“
2020-07-19 14:35:45
364
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人