Python 爬取异步加载的数据

Catalog

爬取内容

• 下拉菜单异步加载的数据
  

获取异步加载的文件

• F12，查看网络，筛选XHR，点击下拉菜单，等待异步加载的文件
  

代码

import requests
from lxml import etree
# 浏览器伪装
ua = 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko'
header = {"User-Agent": ua}
# GET请求
url = 'http://www.runoob.com/ajax/ajax-database.html'
r = requests.get(url, headers=header)
r.encoding = 'urf-8'
text = r.text
# xpath提取信息
element = etree.HTML(text)
options = element.xpath('/html/body/div[3]/div/div[2]/div/div[3]/div/div[1]/div/form/select/option/@value')
print(options)
# 提取异步加载的信息
for option in options:
    # POST请求
    d = {'q': option.strip()}
    url = 'http://www.runoob.com/try/ajax/getcustomer.php'
    r = requests.post(url, headers=header, data=d)
    r.encoding = 'GBK'
    text = r.text
    # xpath提取信息
    element = etree.HTML(text)
    em = element.xpath('//tr/td[1]/em/text()')
    td = element.xpath('//tr/td[2]/text()')
    for e, t in zip(em, td):
        print(e, ' | ', t)
    print('-' * 55, options.index(option) + 1)

打印结果

[‘APPLE’, ‘BAIDU ‘, ‘Canon’, ‘Google’, ‘Nokia’, ‘SONY’]
CustomerID | APPLE
CompanyName | Apple Computer, Inc.
ContactName | Steven Jobs
Address | 1 Infinite Loop Cupertino, CA 95014
City | Cupertino
PostalCode | 95014
Country | USA
——————————————————- 1
CustomerID | BAIDU
CompanyName | BAIDU, Inc
ContactName | Li YanHong
Address | Lixiang guoji dasha,No 58, beisihuanxilu
City | Beijing
PostalCode | 100080
Country | China
——————————————————- 2
CustomerID | Canon
CompanyName | Canon USA, Inc.
ContactName | Tsuneji Uchida
Address | One Canon Plaza Lake Success, NY 11042
City | New York
PostalCode | 11042
Country | USA
——————————————————- 3
CustomerID | Google
CompanyName | Google, Inc.
ContactName | Larry Page
Address | 1600 Amphitheatre Parkway Mountain View, CA 94043
City | Mountain View
PostalCode | 94043
Country | USA
——————————————————- 4
CustomerID | Nokia
CompanyName | Nokia Corporation
ContactName | Olli-Pekka Kallasvuo
Address | P.O. Box 226, FIN-00045 Nokia Group, Finland
City | Helsinki
PostalCode | none
Country | Finland
——————————————————- 5
CustomerID | SONY
CompanyName | Sony Corporation of America
ContactName | Kazuo Hirai
Address | Park Ridge, NJ 07656
City | Park Ridge
PostalCode | 07656
Country | USA
——————————————————- 6