python get header repsonse high performance

最新推荐文章于 2025-08-15 08:27:29 发布

iteye_20379

最新推荐文章于 2025-08-15 08:27:29 发布

阅读量97

点赞数

CC 4.0 BY-SA版权

文章标签： python

本文链接：https://blog.youkuaiyun.com/iteye_20379/article/details/82129607

本文介绍了一个脚本，使用多线程技术并利用HTTP头部请求来批量检查网页链接的状态码，确保链接可用性。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >


from urlparse import urlparse
from threading import Thread
import httplib, sys
from Queue import Queue

concurrent = 200

def doWork():
    while True:
        url=q.get()
        status,url=getStatus(url)
        doSomethingWithResult(status,url)
        q.task_done()

def getStatus(ourl):
    try:
        url = urlparse(ourl)
        conn = httplib.HTTPConnection(url.netloc)   
        conn.request("HEAD", url.path)
        res = conn.getresponse()
        return res.status, ourl
    except:
        return "error", ourl

def doSomethingWithResult(status, url):
    print status, url

q=Queue(concurrent*2)
for i in range(concurrent):
    t=Thread(target=doWork)
    t.daemon=True
    t.start()
try:
    for url in open('urllist.txt'):
        q.put(url.strip())
    q.join()
except KeyboardInterrupt:
    sys.exit(1)