Spider学习笔记（二）:抓取51招聘城市信息

最新推荐文章于 2024-05-31 21:24:28 发布

原创最新推荐文章于 2024-05-31 21:24:28 发布 · 267 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#Spider #爬虫

Spider 专栏收录该内容

14 篇文章

订阅专栏

本文介绍了一种使用Python的requests和lxml库从51job网站爬取全国城市列表的方法。通过发送HTTP请求获取网页内容，再利用XPath解析HTML，最终提取出各城市的名称及其对应的URL。

import requests
import lxml
from lxml import etree

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"
}

cityUrl = 'https://jobs.51job.com/'

def getCity(url):
    response = requests.get(url,headers=headers)

    html = response.content.decode('gbk')

    mytree = lxml.etree.HTML(html)

    cityList = mytree.xpath('/html/body/div[2]/div[2]/div[2]/div[1]/a')
    for city in cityList:
        cityName = city.xpath('./text()')[0]
        cityUrl = city.xpath('./@href')[0]
        print(cityName,cityUrl)

if __name__ == '__main__':

    getCity(cityUrl)