Python抓需要输入用户名密码的https网页

原创已于 2022-03-10 16:14:00 修改 · 7.7k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#python #开发语言

于 2013-07-30 14:11:09 首次发布

C++/Python/java 专栏收录该内容

12 篇文章

订阅专栏

本文介绍了如何使用Python的urllib2库抓取需要HTTP基本认证（用户名和密码）的HTTPS网页内容。通过创建HTTPPasswordMgrWithDefaultRealm和HTTPDigestAuthHandler处理登录信息，并使用opener打开和保存网页内容。

import urllib2
import locale

import sys

__g_codeset = sys.getdefaultencoding()
if "ascii"==__g_codeset:
__g_codeset = locale.getdefaultlocale()[1]

def utf8_to_mbs(s):
    return s.decode("utf-8").encode(__g_codeset)
#
def mbs_to_utf8(s):
    return s.decode(__g_codeset).encode("utf-8")
#
def _queryRows(cu,sql):
    try:
        cu.execute(sql)
        rows = []
        for row in cu:
            rows.append(row)
        #
        return rows
    except:
        print(traceback.format_exc())
    #end
#

#http
response = urllib2.urlopen('http://python.org/')
html = response.read()
print html
#https
url = "https://www.xxx.com/xxx/xxx"
# Add username and password.
username = "scott"
password = "tiger"
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, url, username, password)
handler = urllib2.HTTPDigestAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
# use the opener to fetch a URL
urllib2.install_opener(opener)
response = urllib2.urlopen(url)
file = open("a.html", "w")
file.write(response.read())
file.close()