XXL-job如何从数据库拿取数据

Notail^0^

已于 2022-03-31 10:56:49 修改

阅读量2.1k

点赞数

分类专栏：架构之道---中间件文章标签： spring xxl-job

于 2021-08-16 15:42:25 首次发布

此文章来自李晨璐专栏，转载请声明

本文链接：https://blog.youkuaiyun.com/lclcsdnblink/article/details/119646838

版权

架构之道---中间件专栏收录该内容

6 篇文章

订阅专栏

本文介绍JobScheduleHelper与JobTriggerPoolHelper在XXL-Job中的协作，涉及数据库操作、定时任务调度、线程池设计及任务执行。通过行锁确保并发安全，讲解了start(), refreshNextValidTime(), pushTimeRing()方法的工作原理。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

前提：

首先我们要了解JobScheduleHelper和JobTriggerPoolHelper这两个类

JobScheduleHelper：从数据库按照一定的规则去拿定时任务的数据

JobTriggerPoolHelper：执行具体任务

JobScheduleHelper

这个类是从数据库按照一定规则去拿定时任务的数据，如下图

下面我们看代码，里面有四个方法，我们只关注三个start()，refreshNextValidTime()，pushTimeRing()

start()：开启一个线程从数据库按照一定规则拿到数据

refreshNextValidTime()：更新数据库（三个字段：上一个，下一个触发时间和状态，可根据自己的业务进行拓展）

pushTimeRing()：将执行时间超前5s内放到map中，由ringThread线程去执行

start()

三个重点部分：

用数据库行锁来实现调度中心多节点部署，保证触发器的名称和执行时间相同，则只且仅有一个调度中心节点去下发任务给执行器。
里面两个for循环，根据规则不断拿数据，不断更新数据
两个线程scheduleThread（执行时间过期超过5秒和小于5秒）和ringThread（执行时间超前5秒），这里涉及到ringThread如何精准执行

public class JobScheduleHelper {
    private static Logger logger = LoggerFactory.getLogger(JobScheduleHelper.class);


    private static JobScheduleHelper instance = new JobScheduleHelper();

    public static JobScheduleHelper getInstance() {
        return instance;
    }

    public static final long PRE_READ_MS = 5000;    // pre read

    private Thread scheduleThread;
    private Thread ringThread;
    private volatile boolean scheduleThreadToStop = false;
    private volatile boolean ringThreadToStop = false;
    private volatile static Map<Integer, List<QuartzJob>> ringData = new ConcurrentHashMap<>();


    public void start() {

        scheduleThread = new Thread(new Runnable() {
            @Override
            public void run() {

                try {
                    TimeUnit.MILLISECONDS.sleep(5000 - System.currentTimeMillis() % 1000);
                } catch (InterruptedException e) {
                    if (!scheduleThreadToStop) {
                        Logit.errorLog(e.getMessage(), e);
                    }
                }
                Logit.debugLog(">>>>>>>>> init xxl-job admin scheduler success.");

                // 快慢线程池的最大线程数（getTriggerPoolFastMax= 200，getTriggerPoolSlowMax=100），最终等于6000
                // 快慢线程池的最大线程数*触发qps（每次触发消耗50毫秒，qps=1000/50=20）
                int preReadCount = (XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax() + XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax()) * 20;

                while (!scheduleThreadToStop) {

                    // Scan Job
                    long start = System.currentTimeMillis();

                    Connection conn = null;
                    Boolean connAutoCommit = null;
                    PreparedStatement preparedStatement = null;

                    boolean preReadSuc = true;
                    try {

                        // 调度中心支持多节点部署，基于数据库行锁，保证触发器的名称和执行时间相同，则只且仅有一个调度中心节点去下发任务给执行器。
                        conn = XxlJobAdminConfig.getAdminConfig().getDataSource().getConnection();
                        connAutoCommit = conn.getAutoCommit();
                        conn.setAutoCommit(false);

                        preparedStatement = conn.prepareStatement("select * from quartz_lock where lock_name = 'schedule_lock' for update");
                        preparedStatement.execute();

                        // 1、pre read
                        long nowTime = System.currentTimeMillis();
                        List<QuartzJob> quartzJobList = XxlJobAdminConfig.getAdminConfig().getQuartzJobMapper().scheduleJobQuery(nowTime + PRE_READ_MS, preReadCount);
                        if (quartzJobList != null && quartzJobList.size() > 0) {
                            // 2、push time-ring
                            for (QuartzJob quartzJob : quartzJobList) {

                                // 执行时间过期超过5秒，此次不执行，计算出下次触发时间并持久化
                                if (nowTime > quartzJob.getTriggerNextTime() + PRE_READ_MS) {
                                    // 2.1、trigger-expire > 5s：pass && make next-trigger-time
                                    Logit.warnLog(">>>>>>>>>>> xxl-job, schedule misfire, jobId = " + quartzJob.getJobId());

                                    // 刷新数据库中下次和上次执行时间
                                    refreshNextValidTime(quartzJob, new Date());

                                    // 执行时间过期小于5秒，立即触发一次，并计算下次执行时间
                                } else if (nowTime > quartzJob.getTriggerNextTime()) {

                                    // 触发一次，此处可以设置为自己自定义的执行方法
                                   JobTriggerPoolHelper.trigger(jobInfo.getId(), TriggerTypeEnum.CRON, -1, null, null);

                                    Logit.debugLog(">>>>>>>>>>> xxl-job, schedule push trigger : jobId = " + quartzJob.getJobId());

                                    // 刷新数据库中下次和上次执行时间
                                    refreshNextValidTime(quartzJob, new Date());

                                    // 下次执行时间比当前时间在5s内，预读到缓存ringData中
                                    if (quartzJob.getDeleteStatus() == 0 && nowTime + PRE_READ_MS > quartzJob.getTriggerNextTime()) {

                                        // 1、make ring second
                                        int ringSecond = (int) ((quartzJob.getTriggerNextTime() / 1000) % 60);

                                        // 2、push time ring
                                        pushTimeRing(ringSecond, quartzJob);

                                        // 3、fresh next
                                        refreshNextValidTime(quartzJob, new Date(quartzJob.getTriggerNextTime()));
                                    }

                                } else {
                                    // 执行时间超前5秒，未过期，加入到预读缓存中
                                    int ringSecond = (int) ((quartzJob.getTriggerNextTime() / 1000) % 60);

                                    // 2、push time ring
                                    pushTimeRing(ringSecond, quartzJob);

                                    // 刷新数据库中下次和上次执行时间
                                    refreshNextValidTime(quartzJob, new Date(quartzJob.getTriggerNextTime()));
                                }
                            }

                            // 3、更新数据库
                            for (QuartzJob quartzJob : quartzJobList) {
                                XxlJobAdminConfig.getAdminConfig().getQuartzJobMapper().scheduleUpdate(quartzJob);
                            }

                        } else {
                            preReadSuc = false;
                        }

                        // tx stop


                    } catch (Exception e) {
                        if (!scheduleThreadToStop) {
                            Logit.errorLog(">>>>>>>>>>> xxl-job, JobScheduleHelper#scheduleThread error:{}", e);
                        }
                    } finally {

                        // commit
                        if (conn != null) {
                            try {
                                conn.commit();
                            } catch (SQLException e) {
                                if (!scheduleThreadToStop) {
                                    Logit.errorLog(e.getMessage(), e);
                                }
                            }
                            try {
                                conn.setAutoCommit(connAutoCommit);
                            } catch (SQLException e) {
                                if (!scheduleThreadToStop) {
                                    Logit.errorLog(e.getMessage(), e);
                                }
                            }
                            try {
                                conn.close();
                            } catch (SQLException e) {
                                if (!scheduleThreadToStop) {
                                    Logit.errorLog(e.getMessage(), e);
                                }
                            }
                        }

                        // close PreparedStatement
                        if (null != preparedStatement) {
                            try {
                                preparedStatement.close();
                            } catch (SQLException e) {
                                if (!scheduleThreadToStop) {
                                    Logit.errorLog(e.getMessage(), e);
                                }
                            }
                        }
                    }
                    long cost = System.currentTimeMillis() - start;


                    // Wait seconds, align second
                    if (cost < 1000) {  // scan-overtime, not wait
                        try {
                            // pre-read period: success > scan each second; fail > skip this period;
                            TimeUnit.MILLISECONDS.sleep((preReadSuc ? 1000 : PRE_READ_MS) - System.currentTimeMillis() % 1000);
                        } catch (InterruptedException e) {
                            if (!scheduleThreadToStop) {
                                Logit.errorLog(e.getMessage(), e);
                            }
                        }
                    }

                }
                Logit.debugLog(">>>>>>>>>>> xxl-job, JobScheduleHelper#scheduleThread stop");
            }
        });
        scheduleThread.setDaemon(true);
        scheduleThread.setName("xxl-job, admin JobScheduleHelper#scheduleThread");
        scheduleThread.start();


        // ring thread
        ringThread = new Thread(new Runnable() {
            @Override
            public void run() {

                // align second
                try {
                    TimeUnit.MILLISECONDS.sleep(1000 - System.currentTimeMillis() % 1000);
                } catch (InterruptedException e) {
                    if (!ringThreadToStop) {
                        Logit.errorLog(e.getMessage(), e);
                    }
                }

                while (!ringThreadToStop) {

                    try {
                        // second data
                        List<QuartzJob> ringItemData = new ArrayList<>();
                        int nowSecond = Calendar.getInstance().get(Calendar.SECOND);   // 避免处理耗时太长，跨过刻度，向前校验一个刻度；
                        for (int i = 0; i < 2; i++) {
                            List<QuartzJob> tmpData = ringData.remove((nowSecond + 60 - i) % 60);
                            if (tmpData != null) {
                                ringItemData.addAll(tmpData);
                            }
                        }

                        // ring trigger
                        Logit.debugLog(">>>>>>>>>>> xxl-job, time-ring beat : " + nowSecond + " = " + Arrays.asList(ringItemData));
                        if (ringItemData.size() > 0) {
                            // do trigger
                            for (QuartzJob quartzJob : ringItemData) {
                                // do trigger
                                excute(quartzJob);
                            }
                            // clear
                            ringItemData.clear();
                        }
                    } catch (Exception e) {
                        if (!ringThreadToStop) {
                            Logit.errorLog(">>>>>>>>>>> xxl-job, JobScheduleHelper#ringThread error:{}", e);
                        }
                    }

                    // next second, align second
                    try {
                        TimeUnit.MILLISECONDS.sleep(1000 - System.currentTimeMillis() % 1000);
                    } catch (InterruptedException e) {
                        if (!ringThreadToStop) {
                            Logit.errorLog(e.getMessage(), e);
                        }
                    }
                }
                Logit.debugLog(">>>>>>>>>>> xxl-job, JobScheduleHelper#ringThread stop");
            }
        });
        ringThread.setDaemon(true);
        ringThread.setName("xxl-job, admin JobScheduleHelper#ringThread");
        ringThread.start();
    }
}

refreshNextValidTime

    private void refreshNextValidTime(QuartzJob quartzJob, Date fromTime) throws ParseException {
        // 计算下一次触发时间
        Date nextValidTime = new CronExpression(quartzJob.getJobRule()).getNextValidTimeAfter(fromTime);
        if (nextValidTime != null) {
            quartzJob.setTriggerLastTime(quartzJob.getTriggerNextTime());
            quartzJob.setTriggerNextTime(nextValidTime.getTime());
        } else {
            quartzJob.setJobStatus(1);
            quartzJob.setTriggerLastTime(0);
            quartzJob.setTriggerNextTime(0);
        }
    }

pushTimeRing

    private void pushTimeRing(int ringSecond, QuartzJob quartzJob) {
        // 向map中推数据
        List<QuartzJob> ringItemData = ringData.get(ringSecond);
        if (ringItemData == null) {
            ringItemData = new ArrayList<QuartzJob>();
            ringData.put(ringSecond, ringItemData);
        }
        ringItemData.add(quartzJob);

        Logit.debugLog(">>>>>>>>>>> xxl-job, schedule push time-ring : " + ringSecond + " = " + Arrays.asList(ringItemData));
    }

JobTriggerPoolHelper

重点：

两个线程池：快，慢线程池
线程池执行我们具体的定时任务的业务

快慢线程池

  public void start(){
        fastTriggerPool = new ThreadPoolExecutor(
                10,  // 核心线程数
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolFastMax(),  //最大线程数 200
                60L,  // 空闲时间
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(1000),  // 先进先出的阻塞队列，存放任务，这个队列只能用于被 execute 调用的 Runnable 任务。
                new ThreadFactory() {  // 创建新线程使用的工厂。
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-" + r.hashCode());
                    }
                });

        slowTriggerPool = new ThreadPoolExecutor(
                10,
                XxlJobAdminConfig.getAdminConfig().getTriggerPoolSlowMax(),
                60L,
                TimeUnit.SECONDS,
                new LinkedBlockingQueue<Runnable>(2000),
                new ThreadFactory() {
                    @Override
                    public Thread newThread(Runnable r) {
                        return new Thread(r, "xxl-job, admin JobTriggerPoolHelper-slowTriggerPool-" + r.hashCode());
                    }
                });
    }