Python3-爬取页面元素的几种方法

最新推荐文章于 2024-05-18 17:30:45 发布

原创

最新推荐文章于 2024-05-18 17:30:45 发布 · 1.8k 阅读

6 ·

CC 4.0 BY-SA版权

文章标签：

#Python3 #爬虫 #xpath #BeautifulSoup #正则

本文介绍了Python3中四种爬取网页元素的方法：使用XPath、正则表达式、BeautifulSoup的find和find_all以及select方法。提供了匹配URL的示例和相关资源链接。

方法一：xpath

from lxml import etree
import requests
from requests import exceptions

def get_response(url, headers=None, timeout=None):

    """"
    当条件为假，断言失败
    assert response.status_code == 200'请求报错，请求状态码为: %s'%str(response.status_code)
    """
    try:
        response = requests.get(url, headers=headers, timeout=timeout)
    except exceptions.Timeout as e:
        response = None
        raise e
    except exceptions.HTTPError as e:
        response = None
        raise e
    except exceptions.ConnectTimeout as e:
        response = None
        raise e
    except exceptions.ReadTimeout as e:
        response = None
        raise e
    except exceptions.ProxyError as e:
        response = None
        raise e
    except Exception as e:
        response = None
        raise e
    finally:
        return response

def get_content(etree_html, xpath):
    result = []
    content = etree_html.xpath(xpath)  # list 类型
    for ea