【requests】连接url并下载_20180629

最新推荐文章于 2025-02-24 20:54:31 发布

seabreeze912

最新推荐文章于 2025-02-24 20:54:31 发布

阅读量1.6k

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/seabreeze912/article/details/80861637

本文介绍了一个使用Python的requests库进行网页抓取的例子，包括设置请求头、使用代理、POST请求方式发送数据、解析HTML页面等内容。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

模块：

import requests 
import  json
import re
import bs4

代码：

指定url：

url = 'http://s.taobao.com/search'

指定headers：

headers = {
        'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36',
        'Referer':'http://music.163.com/song?id=4466775',
        'Content-Type':'application/x-www-form-urlencoded'
        }

指定data，用于post模式：

params ='YjWOL6GTfeRlUM7HatfrY/nBu4u1towcIzpATPPS9MidyVxJdCiIFrN0xMqWCKUKLyLwQxOPI7QxHtYeLzcmpMFt9JRpB5fjJBwFW2AGo9avnK2oMkw+VRPy+qOf6lH2kNpBlCrM2CSUfHwntHLzcZFfigEYGJovFYg4nQc1EWDB6z+qrh1TpD4JUrHr6Ya4'
encSecKey = '68bb0b091cc915e90428b1f74d63c4729f9f0cf143b300be9a088a6f05dae177885c914cf1709e845625f698e9dc7c4d3aca61cc2c42721113e7bf16f0684384220fe38c0c2f002c9f756d8d94a854c4c0e5e6b959f7ede3ba8f771f5eb2b181720c180192dd85c7a643d5561f1c7f195faff6b397451ad3110a3c598e133744'

data = {'params':params,'encSecKey':encSecKey}

指定代理IP：

proxies = { "http": "10.10.1.10:3128",  "https": "10.10.1.10:1080"}

获取网页内容：

Get模式：

res = requests.get(url, headers = headers, proxies = proxies)

Post模式：

res = requests.post(url, headers=headers, data=data, proxies = proxies)

写入.txt：

with open('items.txt','w',encoding = 'utf-8') as file:
        file.write(res.text)

读取并正则查找：

with open('items.txt','r',encoding='utf-8') as file1:
        g_page_config = re.search(r'g_page_config = (.*?);\n',file1.read())
        with open('g_page_config.txt','w',encoding='utf-8') as file2:
            file2.write(g_page_config.group(1))

获取bs4模式下的网页：

soup = bs4.BeautifulSoup(res.text,'html.parser')

查找对应的内容：

depth = soup.find('span',class_='next').previous_sibling.previous_sibling.text

targets = soup.find_all('div',class_='hd')
for each in targets:
        movies.append(each.a.span.text)

转化json模式：

comments_json = json.loads(res.text)

下载图片：

url = 'http://placekitten.com/200/300'
res = requests.get(url)

with open('cat.jpg','wb') as f:
    f.write(res.content)

或者jpeg , png

链接模块：

def getHTMLText(url):
    try:
        r = requests.get(url,timeout = 30)
        r.raise_for_status() #如果返回状态不是200，引发HTTPError异常
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return '产生异常'