chapter08_线程池的使用_4_扩展ThreadPoolExecutor

本文介绍如何通过扩展ThreadPoolExecutor类中的hook方法,实现对线程执行前后及线程池关闭时的操作。具体展示了TimingThreadPool类的实现,该类记录了每个任务的执行时间和平均执行时间。
  • ThreadPoolExecutor类中提供了几个hook方法

      class ThreadPoolExecutor {
    
          ...
    
          protected void beforeExecute(Thread t, Runnable r) { }
    
          protected void afterExecute(Runnable r, Throwable t) { }
    
          protected void terminated() { }
    
          ...
      }
    

    所以这几个方法可以自己扩展

    beforeExecute是在执行任务前被调用;aferExecute在执行任务后被调用(无论是否抛出异常都会执行);terminated在线程池关闭时调用

      public class TimingThreadPool extends ThreadPoolExecutor {
    
          public TimingThreadPool() {
    
              super(1, 1, 0L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>());
          }
    
          private final ThreadLocal<Long> startTime = new ThreadLocal<Long>();
          private final Logger log = Logger.getLogger("TimingThreadPool");
          private final AtomicLong numTasks = new AtomicLong();
          private final AtomicLong totalTime = new AtomicLong();
    
          protected void beforeExecute(Thread t, Runnable r) {
    
              super.beforeExecute(t, r);
              log.fine(String.format("Thread %s: start %s", t, r));
              startTime.set(System.nanoTime());
          }
    
          protected void afterExecute(Runnable r, Throwable t) {
    
              try {
                  long endTime = System.nanoTime();
                  long taskTime = endTime - startTime.get();
                  numTasks.incrementAndGet();
                  totalTime.addAndGet(taskTime);
                  log.fine(String.format("Thread %s: end %s, time=%dns", t, r, taskTime));
              } finally {
                  super.afterExecute(r, t);
              }
          }
    
          protected void terminated() {
    
              try {
                  log.info(String.format("Terminated: avg time=%dns", totalTime.get() / numTasks.get()));
              } finally {
                  super.terminated();
              }
          }
      }
    
import requests from lxml import etree import re import pymysql from time import sleep from concurrent.futures import ThreadPoolExecutor def get_conn(): # 创建连接 conn = pymysql.connect(host="127.0.0.1", user="root", password="root", db="novels", charset="utf8") # 创建游标 cursor = conn.cursor() return conn, cursor def close_conn(conn, cursor): cursor.close() conn.close() def get_xpath_resp(url): headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'} resp = requests.get(url, headers=headers) tree = etree.HTML(resp.text) # 用etree解析html return tree,resp def get_chapters(url): tree,_ = get_xpath_resp(url) # 获取小说名字 novel_name = tree.xpath('//*[@id="info"]/h1/text()')[0] # 获取小说数据节点 dds = tree.xpath('/html/body/div[4]/dl/dd') title_list = [] link_list = [] for d in dds[:15]: title = d.xpath('./a/text()')[0] # 章节标题 title_list.append(title) link = d.xpath('./a/@href')[0] # 章节链接 chapter_url = url +link # 构造完整链接 link_list.append(chapter_url) return title_list,link_list,novel_name def get_content(novel_name,title,url): try: cursor = None conn = None conn, cursor = get_conn() # 插入数据的sql sql = 'INSERT INTO novel(novel_name,chapter_name,content) VALUES(%s,%s,%s)' tree,resp = get_xpath_resp(url) # 获取内容 content = re.findall('<div id="content">(.*?)</div>',resp.text)[0] # 对内容进行清洗 content = content.replace('<br />','\n').replace(' ',' ').replace('全本小说网 www.qb5.tw,最快更新<a href="https://www.qb5.tw/book_116659/">宇宙职业选手</a>最新章节!<br><br>','') print(title,content) cursor.execute(sql,[novel_name,title,content]) # 插入数据 conn.commit() # 提交事务保存数据 except: pass finally: sleep(2) close_conn(conn, cursor) # 关闭数据库 if __name__ == '__main__': # 获取小说名字,标题链接,章节名称 title_list, link_list, novel_name = get_chapters('https://www.qb5.tw/book_116659/') with ThreadPoolExecutor(5) as t: # 创建5个线程 for title,link in zip(title_list,link_list): t.submit(get_content, novel_name,title,link) # 启动线程
05-16
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值