python爬虫之全国城市

一起进步吧！感谢大家的支持和关注

需求

爬取热门城市和全部城市的名称
目标地址：https://www.aqistudy.cn/historydata/

---

分析

• 静态页面
• get请求
• 定位 xpath
• 管道符的使用

---

源代码

# 第一种写法
from lxml import etree
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'
}
url = 'https://www.aqistudy.cn/historydata/'
page_text = requests.get(url=url, headers=headers).text
tree = etree.HTML(page_text)

# 解析热门城市
hot_li_list = tree.xpath('//div[@class="bottom"]/ul/li')

for li in hot_li_list:
    city_name = li.xpath('./a/text()')[0]
    with open("hot_city.txt", "a", encoding="utf-8") as f:
        text = f"{city_name}\n"
        f.write(text)

# 解析全部城市
all_li_list = tree.xpath('//div[@class="bottom"]/ul/div[2]/li')
for li in all_li_list:
    city_name = li.xpath('./a/text()')[0]
    with open("all_city.txt", "a", encoding="utf-8") as f:
        text = f"{city_name}\n"
        f.write(text)

# 解析热门城市+所有城市
# 此处xpath表达式的管道符（|）可以是的xpath表达式更加具有通用性
# li_list = tree.xpath('//div[@class="bottom"]/ul/li | //div[@class="bottom"]/ul/div[2]/li')
print(200)

---