2.用scrapy 爬取链家网站 全国的二手房信息。

本文介绍了一款针对链家网站二手房信息的爬虫程序,通过Scrapy框架实现,能够抓取各城市区域的房源详细信息,包括标题、地址、房屋详情、楼层、总价和单价等,并展示了如何设置请求头、解析网页结构及翻页操作。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

# endcoding:utf-8
import scrapy,time,random,re
要使用该爬虫在命令行输入以下命令即可。
# scrapy runspider quotes_spider.py -o quotes.json 
class QuotesSpider(scrapy.Spider):
	# allowed_domains='lianjia.com'
	name = 'lianjia_ershou'
	start_urls=['https://www.lianjia.com/city/']
	headers={
	'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'
	}
	def start_requests(self):
		for url in self.start_urls:
			yield scrapy.Request(url,callback=self.parse_city,headers=self.headers)

	def parse_city(self,response):
		urls=response.css('.city_list a::attr("href")').getall()
		urls=[url+'ershoufang/' for url in urls]
		for url in urls:
			yield scrapy.Request(url, self.parse_quyu,headers=self.headers)
	def parse_quyu(self,response):
		# print('=='*50)
		urls=response.xpath('//div[@data-role="ershoufang"]/div[1]/a/@href').getall()
		for url in urls:
			url=response.urljoin(url)
			if url!=response.url:
				print(url)
				yield scrapy.Request(url,self.parse,headers=self.headers)
		
	def parse(self, response):
		
		data=response.xpath('//div[@class="info clear"]')
		titles=data.css('.title a::text').getall()
		address=data.css('.address .houseInfo a::text').getall()
		houseInfo=data.css('.address .houseInfo::text').getall()
		floods=data.css('.flood').xpath('div/text()').getall()

		priceInfos=data.css('.priceInfo')
		totalPrices=[i+'万' for i in data.css('.totalPrice span::text').getall()]
		unitPrices=priceInfos.css('.unitPrice span::text').getall()
		for title,address,houseInfo,flood,totalPrice,unitPrice in zip(titles,address,houseInfo,floods,totalPrices,unitPrices):
			yield{
				'title':title,
				'address':address,
				'houseInfo':houseInfo,
				'flood':flood,
				'totalPrice':totalPrice,
				'unitPrice':unitPrice,
			}

		resp=response.xpath('//div[@class="page-box house-lst-page-box"]')
		try:
			totalPage=resp.re(r'totalPage":(\d+)')[0]
			curPage=resp.re(r'curPage":(\d+)}')[0]
		except Exception as e:
			print(e)
			return 

		# https://sz.lianjia.com/ershoufang/luohuqu/pg2/
		time.sleep((random.random()+0.5)*2)
		if  int(curPage)!=int(totalPage):
			if not 'pg' in response.url:
				url1=response.url+'pg'+str(int(curPage)+1)+'/'
			else:
				# url1=response.url+'pg'+str(int(curPage)+1)+'/'
				url1=re.sub(re.compile(r"\d+", re.S), str(int(curPage)+1), response.url)

			# print(url1)
			# exit()
			yield scrapy.Request(url1,callback=self.parse,headers=self.headers)

 

下面是使用Scrapy-redis爬取二手房房价的代码: 1. 创建一个Scrapy项目 ``` $ scrapy startproject lianjia ``` 2. 修改settings.py配置文件,启用Redis ``` # 启用Redis调度器 SCHEDULER = "scrapy_redis.scheduler.Scheduler" # 启用Redis去重 DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter" # 启用Redis队列 SCHEDULER_QUEUE_CLASS = "scrapy_redis.queue.SpiderPriorityQueue" # 设置Redis连接信息 REDIS_HOST = 'localhost' REDIS_PORT = 6379 ``` 3. 创建一个Spider ``` $ scrapy genspider lianjia_spider lianjia.com ``` 4. 修改lianjia_spider.py ``` # 引入RedisSpider from scrapy_redis.spiders import RedisSpider class LianjiaSpider(RedisSpider): name = 'lianjia_spider' allowed_domains = ['lianjia.com'] # 设定爬取的初始接 def start_requests(self): url = 'https://sh.lianjia.com/ershoufang/' yield scrapy.Request(url=url, callback=self.parse) # 解析二手房列表页 def parse(self, response): # 获取下一页接并加入Redis队列 next_page = response.css('div.page-box a.next::attr(href)').get() if next_page is not None: next_page_url = response.urljoin(next_page) self.server.rpush('lianjia_spider:start_urls', next_page_url) # 解析二手房信息 house_list = response.css('ul.sellListContent li.clear') for house in house_list: # 获取房屋名称和接 name = house.css('div.title a::text').get() link = house.css('div.title a::attr(href)').get() # 获取房屋价格 price = house.css('div.totalPrice span::text').get() # 获取房屋面积和户型 area = house.css('div.houseInfo a::text').getall()[1] layout = house.css('div.houseInfo a::text').getall()[0] # 输出房屋信息 print(name, link, price, area, layout) ``` 5. 运行爬虫 ``` $ scrapy runspider lianjia_spider.py ``` 6. 启动Redis队列 ``` $ redis-server ``` 7. 启动Redis Spider ``` $ scrapy runspider lianjia_spider.py ``` 运行后,Scrapy会将初始接加入Redis队列,并从队列中获取接进行爬取。当爬取到列表页时,会将下一页的接加入Redis队列,从而实现了分布式爬取爬取到的房屋信息会输出到控制台。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值