Python3访问网络

转载请注明出处

本文出自Hansion的博客

Python3以后， urllib库和urilib2库合并为 urllib库

urllib2.urlopen() 的写法更改为 urllib.request.urlopen()

urllib2.Request() 的写法更改为 urllib.request.Request()  

访问网络示例

import urllib.parse
import urllib.request
url = 'http://www.baidu.com/s'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {'name' : 'WHY',    
         'location' : 'SDU',    
         'language' : 'Python',
         'ie' : 'utf-8',
         'wd' : 'python' }
headers = { '<a href="https://www.baidu.com/s?wd=User-Agent&tn=44039180_cpr&fenlei=mv6quAkxTZn0IZRqIHckPjm4nH00T1Y3nH79uHT3PWmYmyfsrHIh0ZwV5Hcvrjm3rH6sPfKWUMw85HfYnjn4nH6sgvPsT6KdThsqpZwYTjCEQLGCpyw9Uz4Bmy-bIi4WUvYETgN-TLwGUv3EnHfYPH6vPjbLn1RLPjTzrjDYn0" target="_blank" class="baidu-highlight">User-Agent</a>' : user_agent }
data = urllib.parse.urlencode(values)
req = urllib.request.Request(url+'?'+data)
response = urllib.request.urlopen(req)
the_page = response.read()
print(the_page.decode('UTF8'))

如果要添加headers，第12行就要写成这样

req = urllib.request.Request(url, data, headers)

运行时会报错：TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.


我们还需要在第11行后面加上.encode(encoding='UTF8') 才行

data = urllib.parse.urlencode(values).encode(encoding='UTF8')