python爬虫 xpath使用问题整理

1. module 'lxml' has no attribute 'html'

代码：

import requests
import lxml

web = requests.get('https://www.xxx.com/paihang.html', timeout=7)
selector = lxml.html.fromstring(web)
print(selector)

解决：

import lxml

改为

import lxml.html

2.ValueError: can only parse strings

代码：

web = requests.get('https://www.xxx.com/paihang.html', timeout=7)
selector = lxml.html.fromstring(web)
print(selector)

解决：

selector = lxml.html.fromstring(web)

改为

selector = lxml.html.fromstring(web.text)

3.AttributeError: 'NoneType' object has no attribute 'xpath'

代码：

import requests
from lxml import etree

web = requests.get('https://www.xxx.com/top250/')
web.encoding = 'utf-8'
selector = etree.HTML(web.text)

title = selector.xpath('//head/title/text()')
print(title)

解决：

缺少请求头
web = requests.get('https://www.xxx.com/top250/')

改为：

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0'}
web = requests.get('https://www.xxx.com/top250/', headers=headers)