爬虫程序（一）---读取网页

网页抓取与代理切换

最新推荐文章于 2024-05-27 17:50:54 发布

haichao062

最新推荐文章于 2024-05-27 17:50:54 发布

阅读量2.3k

点赞数

CC 4.0 BY-SA版权

分类专栏： python 爬虫文章标签： download windows 浏览器 url 百度

本文链接：https://blog.youkuaiyun.com/haichao062/article/details/8063122

python 同时被 2 个专栏收录

22 篇文章

订阅专栏

爬虫

4 篇文章

订阅专栏

本文介绍了一种使用Python进行网页抓取的方法，通过切换代理和模拟浏览器来提高抓取的成功率。文中详细展示了如何处理超时错误及HTTP错误，并通过增加等待时间和更换代理的方式来解决这些常见问题。

读取网页时候，如果访问速度过快，会返回个timeout错误（10054），因此要在此做个try，并启用下一个代理。（代理可以百度，http 代理，要有端口号）。同时模拟浏览器，可以防止一些返回错误。

#读取网页函数

def FormatHTML( url ):
    flag = True
    count = 0
    sleep_download_time = 0
    time_out = 10
    fails = 0
    HTTP_num = 0
    HTTP_dl = ['211.142.236.132:80', '118.186.9.21:80', '118.186.9.22:80', '211.142.236.132:80']
    while True:
        if fails >= 3:
            return None
            break
        try:
            print u'========开启代理========='
            opener = urllib2.build_opener( urllib2.ProxyHandler( {'http':HTTP_dl[HTTP_num]} ), urllib2.HTTPHandler( debuglevel = 1 ) )
            urllib2.install_opener( opener )
            while flag:
                try:
                    print u'=========模拟浏览器========='
                    i_headers = {"User-Agent": "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.1) Gecko/20090624 Firefox/3.5", "Referer": 'http://www.baidu.com'}
                    req = urllib2.Request( url, headers = i_headers )
                    time.sleep( sleep_download_time )
                    print u'==========读取网页==========='
                    f = urllib2.urlopen( req, timeout = time_out )
                    flag = False
                except urllib2.HTTPError, e:  
                    if e.code == 404:
                        print 'e.code:' + str( e.code ) 
                        count += 1
                        print 'count=' + str( count )
                        if count >= 4:
                            print 'count==' + str( count )
                            flag = False
                            return None 
                    else:
                        sleep_download_time = sleep_download_time + 2
                        time.sleep( sleep_download_time )
                        count += 1
                        print 'urllib2.HTTPError:' + str( e.code )
                        s = sys.exc_info()
                        print s
                        print "Error '%s' happened on line %d" % ( s[1], s[2].tb_lineno )
                        if count == 10:
                            flag = False
                            return None
                except :
                    print 2
                    sleep_download_time = sleep_download_time + 2
                    time.sleep( sleep_download_time )
                    count += 1
                    print url
                    print u"连接超时!"
                    s = sys.exc_info()
                    print "Error '%s' happened on line %d" % ( s[1], s[2].tb_lineno )
                    if count == 10:
                        flag = False
                        return None          
            reader = BeautifulSoup( f.read() ) 
            print u'==========读取完毕===========' 
            f.close()
            break
        except():
            HTTP_num += 1
            s = sys.exc_info()
            print "Error '%s' happened on line %d" % ( s[1], s[2].tb_lineno )
            fails += 1
            time_out += 5
    return reader