美团人气榜按照[{ } ,{ },{ } ]切片输出

美团新房销售板数据爬取

最新推荐文章于 2022-03-15 23:51:34 发布

原创最新推荐文章于 2022-03-15 23:51:34 发布 · 171 阅读

0 ·

CC 4.0 BY-SA版权

本篇博客介绍了一个用于爬取美团新房销售板详细数据的Python爬虫程序。该程序通过发送GET请求获取指定URL的数据，并解析JSON格式的内容。此外，博客还展示了如何设置User-Agent、处理响应数据及简单的延时操作。

# 仅需修改这个地方https://jn.lianjia.com/ershoufang/pg{}rs/   将jn换成你所在城市的拼写首字母小写
import requests
from lxml import etree
import time
import random
import csv
import requests
import json

class LianjiaSpider(object):
    def __init__(self):
        self.url = "https://mobilenext-web.meituan.com/api/newSalesBoard/getSaleBoardDetail?cityId=96&boardType=9&districtId=0&cateId=1&offset=0&limit=15&lat=36.526046191159445&lng=122.062217811"

        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.163 Safari/535.1"}

    def get_page(self, url):
        res = requests.get(url=url, headers=self.headers)
        res.encoding = "utf-8"
        html = res.text
       # print(html)
        results = html.replace('{"totalSize":50,"saleBoardDealList":', "").replace("}", "")
        print(results)
      #  for item in json.loads(results):
     #       print(item)


    def main(self):
        time.sleep(random.randint(3, 5))
        url = self.url
        self.get_page(url)


if __name__ == '__main__':
    start = time.time()
    spider = LianjiaSpider()
    spider.main()
    end = time.time()
    print("执行时间：%.2f" % (end - start))