python爬取网页数据实例(一)

最新推荐文章于 2025-01-15 16:13:45 发布

三爷麋了鹿

最新推荐文章于 2025-01-15 16:13:45 发布

阅读量1.4k

点赞数

分类专栏： Python

本文链接：https://blog.youkuaiyun.com/u800820/article/details/86163442

版权

Python 专栏收录该内容

10 篇文章

订阅专栏

本文分享了一段Python代码，演示如何使用urllib2库发送网络请求，lxml库解析HTML结构，以及urlparse库解析URL参数。通过随机选择User-Agent列表中的元素来模拟不同的浏览器，降低被目标网站封禁的风险。代码具体实现了从百度首页跳转到音乐页面，再获取音乐页面中特定元素的过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

# -*- coding:utf-8 -*-
from lxml import etree
import urllib2
import random
import urlparse

# 设置网络代理
proxy_info = {
    'host': '127.0.0.1',
    'port': 8080
}
proxy_support = urllib2.ProxyHandler({'http': '%(host)s:%(port)d' %proxy_info})
openner = urllib2.build_opener(proxy_support)
urllib2.install_opener(openner)
#user_Agent列表
user_agent_list = [
    "Mozilla/5.0(Macintosh;IntelMacOSX10.6;rv:2.0.1)Gecko/20100101Firefox/4.0.1",
    "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)",
    "Opera/9.80(WindowsNT6.1;U;en)Presto/2.8.131Version/11.11",
    "Mozilla/5.0(Macintosh;IntelMacOSX10_7_0)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.56Safari/535.11",
    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",
    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)"
]
ua_header = {'User-Agent': random.choice(user_agent_list)}
url = 'http://www.baidu.com'
request = urllib2.Request(url, headers=ua_header)
try:
    response = urllib2.urlopen(request, timeout=10)
    print(response.getcode())
    result = response.read()
    res_htm = etree.HTML(result)
    tab_inner = res_htm.xpath("//div[@class='s_tab_inner']/*/text()")
    tab_href = res_htm.xpath("//div[@class='s_tab_inner']/*/@href")
    index = 0
    for inner in tab_inner:
        if str(inner.encode('utf-8')) == '音乐':
            index = tab_inner.index(inner)
            index = int(index) - 1

    url_music = str(tab_href[index])
    request = urllib2.Request(url_music, headers=ua_header)
    response = urllib2.urlopen(request, timeout=60)
    print(response.getcode())
    result = response.read()
    res_htm = etree.HTML(result)
    # print etree.tostring(res_htm, encoding='utf-8')
    responsive = res_htm.xpath("//div[@id='responsive']//div[@class='search-info']//a/@href")
    column = urlparse.urlparse(responsive[0])
    param = urlparse.parse_qs(column.query)
    id = param['id'][0]
    print(id)
except:
    print("网络超时,请稍后再试")

先上代码(Talk is cheap. Show me the code)，见上。

本实例使用urllib2请求网页，lxml解析网页结构，urlparse解析请求参数。