[Python]学以致用爬取豆瓣小组租房信息-优快云博客

本文链接：https://blog.youkuaiyun.com/ZingHd/article/details/80590009

因为马上毕业面临着找房子的需求，在知乎上搜索了一下，发现豆瓣小组推荐的比较多。但是豆瓣小组得一个一个看不方便，想起了昨天学的爬虫，正好可以学以致用，开心之余发了条朋友圈，没想到有相同需求的朋友还不少，那我就把源码分享给大家，希望大家都能找到自己想要的好房子。

使用方法

搜索「豆瓣小组」，在豆瓣小组的搜索界面填写需要租房的地点，如图为例输入「成都租房」。

将页面拉倒最下方，点击「更多小组讨论」。

进入界面后，复制网址，这个网址替换掉程序里的url部分即可。

import time    # 时间函数库，包含休眠函数sleep()
import random
import requests ##导入requests
import csv
from bs4 import BeautifulSoup ##导入bs4中的BeautifulSoup
ip_list=[
    '223.241.119.23:18118', '183.30.204.91:9999', '112.115.57.20:3128'
    ]
user_agent_list = [
     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
     "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
     "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
     "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
     "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
     "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
     "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
     "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
     "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
     "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
     "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
    ]
csv_file=open("d://Code_py//爬虫//爬代理//成都女生租房.CSV", 'w',newline='')#你想保存的位置
csv_writer = csv.writer(csv_file)
for i in range(50):#页码 爬取前50页信息
    #需要爬取的网址，用同样的形式替换，复制的网址start=后面的数字掉即可
    url = 'https://www.douban.com/group/chengduhezu/discussion?start='+str(i*25)
    UA = random.choice(user_agent_list)
    headers = {'User-Agent': UA}
    random_ip = random.choice(ip_list)
    proxy = {'http': random_ip}
    try:
        rec = requests.get(url, headers=headers)
        # rec = requests.get(url, headers=headers,proxies=proxy)#需要ip的话打开这个
        Soup = BeautifulSoup(rec.text, 'lxml')
        a_list = Soup.find_all('a',class_="",title=True)
        for a in range(len(a_list)):#
            title = a_list[a]['title']  # 取出a标签的href 属性
            href = a_list[a]['href']
            print(title,"第%d页第%d个"%(i+1,a+1))
            title_1=[]
            title_1.append(title+"第"+str(i+1)+"页第"+str(a+1)+"个")
            title_1.append(href)
            csv_writer.writerows([title_1])
            title_1=[]
            time.sleep(0.8)#休息0.8s 如果出现链接失败，可以把间隔时间加长
    except:
        print ('connect failed',random_ip)
        random_ip = random.choice(ip_list)
csv_file.close()