Python 爬虫proxy

hitcs_fs

于 2020-10-29 15:14:00 发布

阅读量233

点赞数

分类专栏： python 文章标签： python 爬虫

本文链接：https://blog.youkuaiyun.com/qq_41878777/article/details/109354657

版权

python 专栏收录该内容

2 篇文章

订阅专栏

我电脑挂了内个（可能不让说，没过审一次了），想爬下文中的URL，得用代理，反正亲测可行，封不封ip我还不知道

import requests
from bs4 import BeautifulSoup


def crawl_page(url, headers, session, ):
    proxies = {
        "https":"https://127.0.0.1:10809",
        "http": "http://127.0.0.1:10809"
    }
    req = session.get(url,headers=headers,proxies=proxies)
    # print(req.content)
    print(req.text)

if __name__ == '__main__':
    session = requests.session()
    url = "https://google.com"
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0",
               "Accept": "*/*"}
    crawl_page(url,headers,session)