进程池与线程池例子（异步调用与回调）

最新推荐文章于 2024-08-05 19:57:48 发布

转载最新推荐文章于 2024-08-05 19:57:48 发布 · 68 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/yaya625202/p/9048497.html

本文介绍了一种使用Python的多线程技术来高效抓取多个网站数据的方法，并展示了如何解析这些数据。通过限制线程池大小，可以有效地控制并发数，避免服务器资源过度消耗。

多线程下载网站数据

from concurrent.futures import ThreadPoolExecutor
import time
import requests
def get(url):
    print("get %s" % url)
    response=requests.get(url)
    time.sleep(3)
    return {"url":url,"content":response.text}
def parse(res):
    res=res.result()
    print("%s parse res is %s" % (res['url'],len(res['content'])))

if __name__=="__main__":
    urls=[
        "http://fanyi.youdao.com/",
        "http://news.baidu.com/",
        "http://news.cctv.com/2018/05/16/ARTIjk8z8DQrpR7ciSzijCgl180516.shtml",
        "http://news.cctv.com/2018/05/16/ARTI43pH6clUqOHJfYaRgHri180516.shtml",
        "http://news.china.com/domestic/945/20180516/32417253.html?newsbaidu",
        "http://money.china.com/"
    ]
    pool=ThreadPoolExecutor(2)
    for url in urls:
        pool.submit(get,url).add_done_callback(parse)

get http://fanyi.youdao.com/
get http://news.baidu.com/
http://fanyi.youdao.com/ parse res is 18720
get http://news.cctv.com/2018/05/16/ARTIjk8z8DQrpR7ciSzijCgl180516.shtml
http://news.baidu.com/ parse res is 74087
get http://news.cctv.com/2018/05/16/ARTI43pH6clUqOHJfYaRgHri180516.shtml
http://news.cctv.com/2018/05/16/ARTIjk8z8DQrpR7ciSzijCgl180516.shtml parse res is 88508
get http://news.china.com/domestic/945/20180516/32417253.html?newsbaidu
http://news.cctv.com/2018/05/16/ARTI43pH6clUqOHJfYaRgHri180516.shtml parse res is 32127
get http://money.china.com/
http://news.china.com/domestic/945/20180516/32417253.html?newsbaidu parse res is 65824
http://money.china.com/ parse res is 63167

Process finished with exit code 0

线程池控制服务端线程数量

from socket import *
from concurrent.futures import ThreadPoolExecutor
def communicate(conn):
    while True:
        try:
            data = conn.recv(1024)
            if not data: break
            conn.send(data.upper())
        except ConnectionResetError:
            break
    conn.close()
def servers(ip,port):
    server = socket(AF_INET, SOCK_STREAM)
    server.bind((ip, port))
    server.listen(5)
    while True:
        conn,addr=server.accept()
        print(addr)
        pool.submit(communicate,conn)
    server.close()
if __name__=="__main__":
    pool=ThreadPoolExecutor(2)
    servers("127.0.0.1",8081)

转载于:https://www.cnblogs.com/yaya625202/p/9048497.html