Day27. 爬虫案例二：链家楼盘

爬虫案例二：链家楼盘

---

---

前言

上个案例的粗糙版，也是可以用吧。

---

一.链家楼盘案例

import requests
from bs4 import BeautifulSoup
import time
import pymongo
from multiprocessing import Pool

# 创建数据库以及相应的表格
client = pymongo.MongoClient('localhost', 27017)
lianjia = client['lianjia']
boss_info = lianjia["boss_info"]


def get_soup(url):
    # url = "https://www.zhipin.com/c101210100/?query=python&page=2"
    headers = {
        "user-agent": "******",
        "cookie": "*****"
    }

    respose = requests.get(url, headers=headers)
    soup = BeautifulSoup(respose.text, "lxml")
    return soup


def get_info(url):
    soup = get_soup(url)
    time.sleep(1)

    infos = soup.select("#main > div > ul > li > a > div > div.title ")
    companys = soup.select("#main > div > ul > li > a > div > div.name")
    address_requests = soup.select("#main > div > ul > li > a > div > div.msg")

    for info, company, address_request in zip(infos, companys, address_requests):
        data = {
            "infos": info.text,
            "companys": company.text,
            "address_requests": address_request.text
        }
        boss_info.insert_one(data)
    print("Insert OK!")


def get_urls():
    query = ["python", "数据分析师", "人工智能", "数据挖掘", "自然语言处理", "推荐算法"]
    pages = range(1, 10)
    urls = []
    for i in query:
        for j in pages:
            url = "https://www.zhipin.com/c101210100/?query={}&page={}".format(i, j)
            urls.append(url)
    return urls


if __name__ == "__main__":
    pool = Pool()
    urls = get_urls()
    pool.map(get_info, (url for url in urls))

---

总结

也是新鲜出炉的，但是相对于上个案例来说，没这么优美，也算是能用吧。

溜了遛了，脑壳疼。Loading（27/100）。。。