jstorm源码阅读汇总(一)

jstorm源码阅读汇总(一)

将最近阅读jstorm的源码笔记汇总一下，主要包括jstorm的task，jstorm网络通讯，jstorm限流部分的代码

jstrorm task

task是storm中任务的实质，也就是业务逻辑的载体，首先Task实现了Runnable接口，那我们大致可以猜到task实际是在一个线程中不断了执行某些程序，看一下重写的run方法

public void run() {
    try {
        taskShutdownDameon = this.execute();
    } catch (Throwable e) {
        LOG.error("init task error", e);
        if (reportErrorDie != null) {
            reportErrorDie.report(e);
        } else {
            throw new RuntimeException(e);
        }
    }
}

run方法实质是对execute方法的一个封装，继续看execute方法

public TaskShutdownDameon execute() throws Exception {
    taskSendTargets = echoToSystemBolt();

    // 创建taskTransfer实例，该实例是用于序列化元组(tuple)并通过netty发送出去的
    taskTransfer = mkTaskSending(workerData);

    // 创建实际的task的执行逻辑的RunnableCallback实例
    RunnableCallback baseExecutor = prepareExecutor();
    setBaseExecutors((BaseExecutors) baseExecutor);

    // 创建业务逻辑的执行线程
    AsyncLoopThread executor_threads = new AsyncLoopThread(baseExecutor, false, Thread.MAX_PRIORITY, true);

    // 创建taskReceiver实例，该实例用于接受从元组(tuple)并解码后将它推入执行队列
    taskReceiver = mkTaskReceiver();

    List<AsyncLoopThread> allThreads = new ArrayList<>();
    allThreads.add(executor_threads);

    LOG.info("Finished loading task " + componentId + ":" + taskId);

    taskShutdownDameon = getShutdown(allThreads, baseExecutor);
    return taskShutdownDameon;
}

该方法返回的是一个TaskShutdownDameon实例，我们先看一下getShutdown方法

public TaskShutdownDameon getShutdown(List<AsyncLoopThread> allThreads, RunnableCallback baseExecutor) {
    AsyncLoopThread ackerThread;
    // 创建ack的执行线程，并添加到线程List里
    if (baseExecutor instanceof SpoutExecutors) {
        ackerThread = ((SpoutExecutors) baseExecutor).getAckerRunnableThread();

        if (ackerThread != null) {
            allThreads.add(ackerThread);
        }
    }
    // 将解码的执行线程添加到线程List里
    List<AsyncLoopThread> recvThreads = taskReceiver.getDeserializeThread();
    for (AsyncLoopThread recvThread : recvThreads) {
        allThreads.add(recvThread);
    }

    // 将编码的执行线程添加到线程List里
    List<AsyncLoopThread> serializeThreads = taskTransfer.getSerializeThreads();
    allThreads.addAll(serializeThreads);
    TaskHeartbeatTrigger taskHeartbeatTrigger = ((BaseExecutors) baseExecutor).getTaskHbTrigger();

    // 创建TaskShutdownDameon线程
    return new TaskShutdownDameon(
            taskStatus, topologyId, taskId, allThreads, zkCluster, taskObj, this, taskHeartbeatTrigger);
}

可以看到该方法主要是将一些藏在其它实例里的线程给加入线程List里，然后构造了一个TaskShutdownDameon实例，该实例应该包含用于清理所有资源的方法，看一下TaskShutdownDameon，该类实现了ShutdownableDameon，而其重写的三个方法最主要的就是shutdown方法，该方法中做了资源的清理

@Override
public void shutdown() {
    if (isClosing.compareAndSet(false, true)) {
        LOG.info("Begin to shut down task " + topologyId + ":" + taskId);

        TopologyContext userContext = task.getUserContext();
        // 清理钩子
        for (ITaskHook iTaskHook : userContext.getHooks())
            iTaskHook.cleanup();
        // 根据task的类型(IBolt/ISpout)调用相应的清理操作
        closeComponent(taskObj);
        // 更新心跳触发器的任务状态
        taskHeartbeatTrigger.updateExecutorStatus(TaskStatus.SHUTDOWN);

        // wait 50ms for executor thread to shutdown to make sure to send shutdown info to TM
        try {
            Thread.sleep(50);
        } catch (InterruptedException ignored) {
        }

        // all thread will check the taskStatus
        // once it has been set to SHUTDOWN, it will quit
        // 更新任务状态
        taskStatus.setStatus(TaskStatus.SHUTDOWN);

        // 将所有线程都进行清理
        for (AsyncLoopThread thr : allThreads) {
            LOG.info("Begin to shutdown " + thr.getThread().getName());
            thr.cleanup();
            JStormUtils.sleepMs(10);
            thr.interrupt();
            // try {
            // //thr.join();
            // thr.getThread().stop(new RuntimeException());
            // } catch (Throwable e) {
            // }
            LOG.info("Successfully shutdown " + thr.getThread().getName());
        }

        taskHeartbeatTrigger.unregister();
        LOG.info("Successfully shutdown task heartbeat trigger for task:{}", taskId);

        try {
            zkCluster.disconnect();
        } catch (Exception e) {
            LOG.error("Failed to disconnect zk for task-" + taskId);
        }

        LOG.info("Successfully shutdown task " + topologyId + ":" + taskId);
    }
}

至此jstorm的task大致也解析完了，接下来看一下task中除了处理业务的外最为重要的其它两个实例——taskTransfer与taskReceiver，看这两个类TaskTransfer和TaskReceiver，这两个类大致是差不多的，重要的都是类内部的两个实现了RunnableCallback的私有类。

• TaskTransfer的内部类

  protected class TransferRun