使用多线程方法加速Python爬虫

1.Python创建多线程的方法

①准备一个函数：

def my_func(a,b):
      do_craw(a,b)

②创建一个线程：

import threading
t=threading.Thread(target=my_func,args=(100,200))

③启动线程：

t.start()

④等待结束：

t.join

例如：

import threading
def work():
    print("working")
for i in range(5):
    t = threading.Thread(target=work)
    t.start()
    print("please wait!")
t.join()
输出结果：
please wait!
working
please wait!
working
please wait!
working
please wait!
working
please wait!
working

Process finished with exit code 0

也可以传递参数:

import threading
def work(i):
    print("%i is working" % i)
for i in range(5):
    t = threading.Thread(target=work, args=(i,))
    t.start()
    print("please wait!")
    t.join()
输出结果：
0 is workingplease wait!

1 is workingplease wait!

2 is workingplease wait!

3 is workingplease wait!

4 is workingplease wait!

每个Thread实例都有一个带有默认值的名字，也可以传入name参数更改。

import threading
import time


def work():
       print(threading.current_thread().getName(), "starting!")
       time.sleep(0.5)
       print(threading.current_thread().getName(), "end!")
t = threading.Thread(name="worker", target=work)
t.start()
print("please wait!")
t.join()
输出结果：
worker starting!
please wait!
worker end!

2.改写爬虫程序，变成多线程爬取

import requests
urls=[
    f"http://www.cnblogs.com/#p{page}"
    for page in range(1,50+1)
]
def craw(url):
    r=requests.get(url)
    print(url,len(r.text))
craw(urls[0])
输出结果：
raise ValueError("check_hostname requires server_hostname")
ValueError: check_hostname requires server_hostname

单线程和多线程的对比：

import blog_spider
import threading
def single_thread():
    for url in blog_spider.urls:
        blog_spider.craw(url)
def multi_thread():
    threads=[]
    for url in blog_spider.urls:
        threads.append(
            threading.Thread(target=blog_spider.craw,args=(url,))
        )
    for thread in threads:
        thread.start()
    for thread in threads:
        thread.join()
if __name__=="__main__":
    start=time.time()
    single_thread()
    end=time.time()
    print("single thread cost:",end-start,"seconds")

3.速度对比：单线程爬虫VS多线程爬虫

在这个例子中，代码改写后，代码快了16倍。

参考资料：
B站《使用多线程，Pyhton爬虫被加速10倍》