前言
从2.0版本开始这个系统将会实现并行执行多个分支的能力，但因为包主的能力有限，不太能纯手敲实现，或只能写成屎山类的，因此这次找到了一个很牛的开源系统，会在此系统上进行二次开发和学习。
开源系统的链接：
https://github.com/icefoxhz/taskflow-java-
这个开源系统算比较复杂的，本着学习的初衷，我还是会先借鉴最核心和基础的并发执行能力，其他的在以后有机会的话可能会参考补充。本篇v2.0会先实现一个简单基础的并行架构，因此建议大佬可以直接跟着链接过去参考了（话说到底有没有非机器人在看这个专栏）。
系统分析
要实现并行执行组件的能力，就不能简单的依靠v1.x中 FlowNode 类的顺序字段来决定执行顺序了，也因此不能简单的通过循环组件流list来执行了。
参考开源代码的思路，可以用dag——有向无环图 作为这个系统的架构，放到实际应用中可以加一个线的类，用于记录组件之间是如何连接的；
然后需要定义一个引擎类，作为计算如何执行这个组件流的引擎，每次调用组件流的时候，流程是创建引擎 --> 解析组件之间关系 --> 执行引擎 --> 得到结果。
引擎的核心逻辑是：
- 如果一个组件依赖多个组件的结果，那么必须等待这些组件全部运行结束，得到结果后才能开始此组件的执行。
- 没有依赖关系的组件，可以用多个线程并行执行。
- 主线程需要阻塞等待组件执行结束
学习的这个开源系统在第一点是通过记录每个组件的入度（indegree）实现，依赖几个组件则入度是几，每执行完成一个组件入度减一，入度归0时可以执行本组件；
第二点通过线程池实现，每次执行最后一个后继节点时用当前线程（如果后继只有一个也是用当前线程），其他的后继节点则创建新线程；
第三点通过CountDownLatch实现，初始值设置为结束节点的个数——因为此开源系统可以定义多个起始和结束节点，但我这里暂时只需要一个即可。同时CountDownLatch的设置最大阻塞时间的功能也很方便，所以这里依然使用这个实现。
代码示例
因为类变多了，很多包我都只把后缀加了个2为了更清晰。
lowcode.core.dto2.Edge
package com.example.lowcode.core.dto2;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.io.Serializable;
/**
* 组件之间的连线
*
* @author llxzdmd
*/
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Edge implements Serializable {
// 起始组件
String sourceName;
// 目标组件
String targetName;
}
lowcode.core.dto2.Flow
这个类是代替v1.x中的list，一个Flow就代表一个可执行的组件流。
package com.example.lowcode.core.dto2;
import com.example.lowcode.core.dto.FlowNode;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.io.Serializable;
import java.util.List;
/**
* 组件流2.0。1.0是直接用List<FlowNode>表示组件流，2.0因为加了连线，用List不合理，所以新建一个更外层的对象代替之前的list
*
* @author llxzdmd
*/
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Flow implements Serializable {
private Long id;
private String name;
private String description;
private String createName;
private String modifiedName;
private List<FlowNode> nodeInstances;
private List<Edge> edgeInstances;
}
lowcode.core.dto2.FlowEngineBuilder
这个类用于解析组件之间的依赖关系
package com.example.lowcode.core.dto2;
import com.example.lowcode.core.dto.ComponentInfo;
import com.example.lowcode.core.dto.FlowNode;
import com.example.lowcode.core.exception.FlowConfigException;
import com.example.lowcode.core.framework.SpringUtil;
import com.example.lowcode.core.framework2.*;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.stream.Collectors;
/**
* @author llxzdmd
*/
@Builder
@Data
@AllArgsConstructor
@NoArgsConstructor
public class FlowEngineBuilder<O> {
private Flow flow;
private Map<String, ComponentInfo> inputParams;
private ExecutorService executor;
public FlowEngineBuilder<O> setFlow(Flow flow) {
this.flow = flow;
return this;
}
public FlowEngineBuilder<O> setExecutor(ExecutorService executor) {
this.executor = executor;
return this;
}
public FlowEngineBuilder<O> setInputParams(Map<String, ComponentInfo> inputParams) {
this.inputParams = inputParams;
return this;
}
public DagEngine<O> buildEngine() {
check();
DagEngine<O> engineWithOpConfig = getEngineWithOpConfig(flow);
clear();
return engineWithOpConfig;
}
private void check() {
if (flow == null) throw new FlowConfigException("Flow不能为null");
if (executor == null) throw new FlowConfigException("线程池不能为null");
}
private void clear() {
this.setFlow(null);
this.setExecutor(null);
this.setInputParams(null);
}
private DagEngine<O> getEngineWithOpConfig(Flow flow) {
DagEngine<O> engine = new DagEngine<>(executor);
// 得到执行类OperatorWrapper列表
List<OperatorWrapper<?, ?>> operatorWrappers = getWrappersWithOpConfig(flow, engine);
// 解析依赖
resolveDependencies(flow, operatorWrappers);
return engine;
}
private List<OperatorWrapper<?, ?>> getWrappersWithOpConfig(Flow flow, DagEngine<O> engine) {
List<OperatorWrapper<?, ?>> operatorWrappers = new ArrayList<>();
for (FlowNode node : flow.getNodeInstances()) {
DefaultInvokeMethodComponent component = getDefaultComponent();
// 创建Op元信息
OpConfig opConfig = new OpConfig();
opConfig.setComponentInfo(ComponentInfo.builder()
.flowId(flow.getId())
.componentId(node.getComponentId())
.inputs(node.getComponentInfo().getInputs())
.outputs(node.getComponentInfo().getOutputs())
.build());
opConfig.setClassName(node.getComponentMetaInfo().getClassName());
// 创建执行类OperatorWrapper
OperatorWrapper<OperatorWrapper<?, ?>, Object> operatorWrapper = new OperatorWrapper<OperatorWrapper<?, ?>, Object>()
.instanceName(node.getNodeName())
.opConfig(opConfig, component)
.engine(engine);
operatorWrappers.add(operatorWrapper);
}
return operatorWrappers;
}
/**
* 解析依赖
*
* @param flow
* @param operatorWrappers
*/
private void resolveDependencies(Flow flow, List<OperatorWrapper<?, ?>> operatorWrappers) {
Map<String, OperatorWrapper<?, ?>> wrapperMap = operatorWrappers.stream()
.collect(Collectors.toMap(OperatorWrapper::getInstanceName, e -> e));
Map<String, List<Edge>> groupBySource = flow.getEdgeInstances().stream().collect(Collectors.groupingBy(
Edge::getSourceName
));
groupBySource.forEach((id, followings) -> {
for (Edge following : followings) {
OperatorWrapper<?, ?> targetOp = wrapperMap.get(following.getTargetName());
targetOp.depend(id);
}
});
}
private DefaultInvokeMethodComponent getDefaultComponent() {
try {
return SpringUtil.getBean(DefaultInvokeMethodComponent.class);
} catch (Exception e) {
throw new FlowConfigException("组件不存在", e);
}
}
}
lowcode.core.dto2.OperatorResult
这个类用于存放执行结果
package com.example.lowcode.core.dto2;
import com.example.lowcode.core.model2.ResultState;
import lombok.Getter;
/**
* @author llxzdmd
*/
@Getter
public class OperatorResult<V> {
/**
* 执行的结果
*/
private V result;
/**
* 结果状态
*/
private ResultState resultState;
/**
* 异常信息
*/
private Throwable ex;
public OperatorResult(V result, ResultState resultState) {
this(result, resultState, null);
}
public OperatorResult(V result, ResultState resultState, Exception ex) {
this.result = result;
this.resultState = resultState;
this.ex = ex;
}
public static <V> OperatorResult<V> defaultResult() {
return new OperatorResult<>(null, ResultState.DEFAULT);
}
@Override
public String toString() {
return "OperatorResult{" +
"result=" + result +
", resultState=" + resultState +
", ex=" + ex +
'}';
}
public void setEx(Throwable ex) {
this.ex = ex;
}
public void setResult(V result) {
this.result = result;
}
public void setResultState(ResultState resultState) {
this.resultState = resultState;
}
}
lowcode.core.framework2.DagEngine
dag引擎类，基本直接用的开源系统的代码，但删了很多东西
package com.example.lowcode.core.framework2;
import com.alibaba.ttl.threadpool.TtlExecutors;
import com.example.lowcode.core.dto2.OperatorResult;
import com.example.lowcode.core.exception.FlowExecutionException;
import com.example.lowcode.core.exception.TaskFlowException;
import com.example.lowcode.core.framework.ComponentContext;
import com.example.lowcode.core.model2.DagWrapperState;
import com.example.lowcode.core.model2.ResultState;
import com.example.lowcode.util.ConvertUtil;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
/**
* DAG执行引擎
*
* @author llxzdmd
*/
@Slf4j
@Getter
public class DagEngine<O> {
/**
* 工作线程池
*/
private ExecutorService executor;
/**
* 主线程阻塞等待所有结束节点执行完成
*/
private CountDownLatch syncLatch;
/**
* wrapper集合
*/
private Map<String, OperatorWrapper<?, ?>> wrapperMap = new HashMap<>();
/**
* 工作线程的快照
*/
private ConcurrentHashMap.KeySetView<Thread, Boolean> runningThreadSet = ConcurrentHashMap.newKeySet();
/**
* OP异常堆栈，只保存一个
*/
private Throwable ex;
/**
* 执行的状态
*/
private int state = DagWrapperState.INIT.getStage();
/**
* 执行引擎上下文
*/
// private DagContext<O> dagContext = new DagContext<O>();
private ComponentContext context;
/**
* 开始节点集合
*/
private Set<OperatorWrapper<?, ?>> beginWrapperSet = new HashSet<>();
/**
* 结束节点集合
* 引擎执行过程中可以根据节点执行情况动态设置结束节点，需要使用线程安全的集合
*/
private Set<OperatorWrapper<?, ?>> endWrapperSet = ConcurrentHashMap.newKeySet();
/**
* 引擎执行结果
*/
private OperatorResult<O> output;
public DagEngine(ExecutorService executor) {
this.executor = TtlExecutors.getTtlExecutorService(executor);
}
/**
* 阻塞主线程，等待流程执行结束，根据依赖关系自动解析出开始节点
*/
public void runAndWait() {
//解析依赖
parseNextDepend();
if (beginWrapperSet == null || beginWrapperSet.isEmpty()) {
throw new TaskFlowException("Flow无开始节点，请检查并修复配置");
}
if (endWrapperSet == null || endWrapperSet.isEmpty()) {
throw new TaskFlowException("Flow无终止节点，请检查并修复配置");
}
OperatorWrapper[] beginWrappers = ConvertUtil.set2Array(beginWrapperSet);
this.runWrappers(beginWrappers);
}
private void runWrappers(OperatorWrapper<?, ?>[] beginWrappers) {
try {
state = DagWrapperState.RUNNING.getStage();
//设置DAG引擎上下文，上下文的生命周期从开始节点到结束节点之间
//DagContextHolder.set(dagContext);
this.context = new ComponentContext();
//初始化信号量
syncLatch = new CountDownLatch(endWrapperSet.size());
//将初始节点交给线程池执行，此过程是异步的
for (OperatorWrapper<?, ?> wrapper : beginWrappers) {
doRun(wrapper, true);
}
//线程阻塞等待DAG执行结束，或超时被唤醒
awaitAndInterruptRunningThread();
} catch (Throwable e) {
log.error("getDagInitialTask error", e);
} finally {
//清理DAG引擎上下文
//DagContextHolder.remove();
}
}
/**
* 主线程阻塞，唤醒后会打断还在执行中的节点
*/
private void awaitAndInterruptRunningThread() {
boolean isTimeout = false;
try {
isTimeout = !syncLatch.await(4, TimeUnit.HOURS);
} catch (InterruptedException e) {
state = DagWrapperState.ERROR.getStage();
log.error("dagEngine is interrupted", e);
}
if (isTimeout) {
state = DagWrapperState.ERROR.getStage();
log.error("DagEngine execution timeout");
final FlowExecutionException flowExecutionTimeout = new FlowExecutionException("Flow execution timeout");
this.ex = flowExecutionTimeout;
throw flowExecutionTimeout;
}
//理论上不会出现这种情况，防御性容错
if (state == DagWrapperState.RUNNING.getStage()) {
state = DagWrapperState.FINISH.getStage();
}
if (!runningThreadSet.isEmpty()) {
for (Thread thread : runningThreadSet) {
thread.interrupt();
}
}
}
/**
* 解析节点之间的依赖关系、开始节点集合、结束节点集合
*/
private void parseNextDepend() {
//解析节点依赖关系
for (Map.Entry<String, OperatorWrapper<?, ?>> entry : wrapperMap.entrySet()) {
OperatorWrapper<?, ?> wrapper = entry.getValue();
//根据 depend 解析依赖关系
Map<String, Boolean> dependWrapperIdMap = wrapper.getDependWrapperIdMap();
if (dependWrapperIdMap != null && !dependWrapperIdMap.isEmpty()) {
for (String dependId : dependWrapperIdMap.keySet()) {
OperatorWrapper<?, ?> depend = wrapperMap.get(dependId);
if (wrapper.getDependWrappers() != null && wrapper.getDependWrappers().contains(depend)) {
continue;
}
//将前驱节点添加到当前节点的依赖集合中
if (wrapper.getDependWrappers() == null) {
wrapper.setDependWrappers(new HashSet<>());
}
wrapper.getDependWrappers().add(depend);
//将当前节点添加到前驱节点的后继集合中
if (depend.getNextWrappers() == null) {
depend.setNextWrappers(new HashSet<>());
}
depend.getNextWrappers().add(wrapper);
//强依赖前驱节点时，indegree+1，弱依赖不需要
if (dependWrapperIdMap.get(dependId)) {
wrapper.getIndegree().incrementAndGet();
if (depend.getSelfIsMustSet() == null) {
depend.setSelfIsMustSet(new HashSet<>());
}
//将当前节点添加到前驱节点的强依赖集合
depend.getSelfIsMustSet().add(wrapper);
}
}
}
}
//解析开始节点、结束节点
for (Map.Entry<String, OperatorWrapper<?, ?>> entry : wrapperMap.entrySet()) {
OperatorWrapper<?, ?> wrapper = entry.getValue();
if (wrapper.getDependWrappers() == null || wrapper.getDependWrappers().isEmpty()) {
beginWrapperSet.add(wrapper);
}
if (wrapper.getNextWrappers() == null || wrapper.getNextWrappers().isEmpty()) {
endWrapperSet.add(wrapper);
}
}
}
/**
* 将节点包装成线程提交给线程池执行，或直接复用当前线程执行
*/
private void doRun(OperatorWrapper<?, ?> wrapper, boolean inNewThread) {
//DAG引擎状态不是RUNNING，说明编排已经执行完
if (state != DagWrapperState.RUNNING.getStage()) {
return;
}
//wrapper节点状态不等于INIT，说明该节点已经执行过
if (!wrapper.compareAndSetState(DagWrapperState.INIT.getStage(), DagWrapperState.RUNNING.getStage())) {
return;
}
if (inNewThread) {
executor.submit(this.getRunningTask(wrapper));
} else {
this.getRunningTask(wrapper).run();
}
}
/**
* 获取执行节点逻辑的线程，主要处理流程逻辑如下：
* 1、将执行线程绑定到当前节点，添加到工作线程集合，打断时使用
* 2、执行全局的节点回调接口
* 3、执行节点主逻辑
* 4、节点执行异常后，停止引擎
* 5、节点正常执行完，根据条件选择后续要执行的分支、节点(组)等
* 6、将节点执行结果保存到上下文、在工作线程集合中删除当前线程、判断是否将信号量减一
* 7、执行后继节点
*/
private Runnable getRunningTask(OperatorWrapper wrapper) {
return () -> {
try {
wrapper.setThread(Thread.currentThread());
runningThreadSet.add(Thread.currentThread());
this.doExecute(wrapper);
wrapper.compareAndSetState(DagWrapperState.RUNNING.getStage(), DagWrapperState.FINISH.getStage());
wrapper.getOperatorResult().setResultState(ResultState.SUCCESS);
} catch (Throwable throwable) {
wrapper.compareAndSetState(DagWrapperState.RUNNING.getStage(), DagWrapperState.ERROR.getStage());
wrapper.getOperatorResult().setResultState(ResultState.EXCEPTION);
wrapper.getOperatorResult().setEx(throwable);
this.ex = throwable;
//当前节点被其它节点强依赖，出现异常中断整个执行流程
if (wrapper.getSelfIsMustSet() != null && !wrapper.getSelfIsMustSet().isEmpty()) {
state = DagWrapperState.ERROR.getStage();
}
} finally {
//将OP的执行结果保存到上下文
//DagContextHolder.putOperatorResult(wrapper.getInstanceName(), wrapper.getOperatorResult());
//节点执行完，释放线程
wrapper.setThread(null);
//从工作线程快照中移除该节点的线程
runningThreadSet.remove(Thread.currentThread());
//如果是结束节点，则将信号量减一
boolean isEndOp = false;
if (endWrapperSet.contains(wrapper)) {
isEndOp = true;
if (endWrapperSet.size() <= syncLatch.getCount()) {
syncLatch.countDown();
}
endWrapperSet.remove(wrapper);
//结束节点全部执行完，停止DAG引擎
if (endWrapperSet.isEmpty()) {
//状态不是 RUNNING，有以下几种情况
//1、多线程情况下，同一时刻多个结束节点都执行完，状态被其它线程修改
//2、当前节点执行异常，已经设置成ERROR状态
//3、引擎执行超时，主线程被唤醒，设置成了ERROR状态
state = state == DagWrapperState.RUNNING.getStage() ? DagWrapperState.FINISH.getStage() : state;
//dagContext.setOutput(wrapper.getOperatorResult());
this.output = wrapper.getOperatorResult();
}
}
if (state != DagWrapperState.RUNNING.getStage()) {
//停止引擎，唤醒阻塞线程
shutdown(state);
}
//通知后继节点
if (!isEndOp) {
notifyNextWrappers(wrapper, wrapper.getNextWrappers());
}
}
};
}
/**
* 调用目标operator的 execute 方法，主要逻辑如下：
* 1、执行节点监听器（开始、异常、成功）
* 2、执行节点回调接口
* 3、执行节点主逻辑
*/
private void doExecute(OperatorWrapper wrapper) throws Exception {
Object operatorResult = wrapper.getOperator().execute(wrapper, context);
wrapper.getOperatorResult().setResult(operatorResult);
context.getContextMap().putAll((Map) operatorResult);
// wrapper.getContext().getContextMap().putAll((Map) operatorResult);
}
/**
* 停止DAG引擎，唤醒阻塞线程
*/
private void shutdown(int currState) {
state = currState;
while (syncLatch.getCount() >= 1) {
syncLatch.countDown();
}
}
/**
* 执行当前节点的后继节点，主要逻辑如下：
* 1、后继节点强依赖当前节点，将入度减一（弱依赖时，不计入入度）
* 2、如果后继节点入度不为零，不执行
* 3、如果后继节点有弱依赖的节点时，后继节点在执行时可以将还在执行中或未执行的依赖节点中断
*/
private void notifyNextWrappers(OperatorWrapper<?, ?> wrapper, Set<OperatorWrapper<?, ?>> nextWrappers) {
if (nextWrappers == null || state != DagWrapperState.RUNNING.getStage()) {
return;
}
Set<OperatorWrapper<?, ?>> selfIsMustSet = wrapper.getSelfIsMustSet();
List<OperatorWrapper<?, ?>> needRunWrappers = null;
for (OperatorWrapper<?, ?> next : nextWrappers) {
//当前节点被后继节点强依赖时，需要将后继节点的 indegree-1
if (selfIsMustSet != null && selfIsMustSet.contains(next)) {
next.getIndegree().decrementAndGet();
}
//强依赖的节点没有执行完时，节点不能开始执行
if (next.getIndegree().get() != 0) {
continue;
}
//节点不能重复执行
if (next.getWrapperState().get() != DagWrapperState.INIT.getStage()) {
continue;
}
if (needRunWrappers == null) {
needRunWrappers = new ArrayList<>();
}
needRunWrappers.add(next);
}
if (needRunWrappers != null) {
for (int i = 0; i < needRunWrappers.size(); i++) {
OperatorWrapper<?, ?> nextWrapper = needRunWrappers.get(i);
boolean inNewThread = true;
//执行最后一个后继节点时复用当前线程，其它的节点交给线程池执行
if (i == needRunWrappers.size() - 1) {
inNewThread = false;
}
doRun(nextWrapper, inNewThread);
}
}
}
}
lowcode.core.framework2.IOperator
这里把执行的方法多抽象了一个接口出来，下面的OperatorWrapper类中记录这个接口的实现，然后再调用具体的执行方法execute。
package com.example.lowcode.core.framework2;
import com.example.lowcode.core.dto2.OperatorResult;
import com.example.lowcode.core.framework.ComponentContext;
/**
* Operator接口
*
* @author llxzdmd
*/
public interface IOperator<P, V> {
/**
* 自定义OP的默认返回值，比如节点执行异常时
*/
default V defaultValue(P param) {
return null;
}
/**
* 该方法实现OP的具体处理逻辑
*/
V execute(P param, ComponentContext context) throws Exception;
}
lowcode.core.framework2.DefaultInvokeMethodComponent
IOperator接口的实现类
package com.example.lowcode.core.framework2;
import com.example.lowcode.core.exception.FlowConfigException;
import com.example.lowcode.core.framework.AbstractComponent;
import com.example.lowcode.core.framework.ComponentContext;
import com.example.lowcode.core.framework.SpringUtil;
import org.springframework.stereotype.Component;
import java.util.Map;
/**
* @author llxzdmd
*/
@Component
public class DefaultInvokeMethodComponent implements IOperator<OperatorWrapper<?, ?>, Object> {
@Override
public Object execute(OperatorWrapper<?, ?> param, ComponentContext context) throws Exception {
return invokeMethodByName(param, context);
}
private Object invokeMethodByName(OperatorWrapper<?, ?> param, ComponentContext context){
OpConfig opConfig = param.getOpConfig();
Map<String, Object> result;
try {
Class<?> aClass = Class.forName(opConfig.getClassName());
AbstractComponent abstractComponent = (AbstractComponent) SpringUtil.getBean(aClass);
result = abstractComponent.execute(context, opConfig.getComponentInfo());
} catch (Exception e) {
throw new RuntimeException(e);
}
return result;
}
}
lowcode.core.framework2.OperatorWrapper
核心类之二，这个类用于执行dag引擎
package com.example.lowcode.core.framework2;
import com.example.lowcode.core.dto2.OperatorResult;
import com.example.lowcode.core.exception.TaskFlowException;
import com.example.lowcode.core.framework.ComponentContext;
import com.example.lowcode.core.model2.DagWrapperState;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
/**
* OP节点的包装类
*
* @author llxzdmd
*/
public class OperatorWrapper<P, V> {
/**
* 该wrapper的id，默认是Operator的全限定名
*/
private String instanceName;
/**
* OP节点配置
*/
private OpConfig opConfig;
/**
* 该wrapper具体要执行的目标OP
*/
private IOperator<P, V> operator;
/**
* 依赖该OP的后续OP集合
*/
private Set<OperatorWrapper<?, ?>> nextWrappers;
/**
* 该OP依赖的OP集合
*/
private Set<OperatorWrapper<?, ?>> dependWrappers;
/**
* 该OP依赖的OP集合id
*/
private Map<String /*dependWrapperId*/, Boolean /*当前节点是否强依赖前置节点*/> dependWrapperIdMap;
/**
* 强依赖于该OP的后续wrapper集合，是nextWrappers的子集
*/
private Set<OperatorWrapper<?, ?>> selfIsMustSet;
/**
* 节点的入度，不同于常规的定义，这里的入度只计算强依赖的节点，当 indegree=0 时，当前OP才能执行，在一个编排流程中，一定满足如下条件
* indegree <= dependWrappers.size()
*/
private AtomicInteger indegree = new AtomicInteger(0);
/**
* OP返回的结果
*/
private volatile OperatorResult<V> operatorResult = OperatorResult.defaultResult();
/**
* 当前节点执行的状态
*/
private AtomicInteger wrapperState = new AtomicInteger(DagWrapperState.INIT.getStage());
/**
* 执行该节点的线程
*/
private Thread thread;
/**
* 绑定的DAG执行引擎
*/
private DagEngine<?> engine;
public OperatorWrapper<P, V> instanceName(String instanceName) {
this.instanceName = instanceName;
return this;
}
public OperatorWrapper<P, V> opConfig(OpConfig opConfig, IOperator<P, V> operator) {
this.opConfig = opConfig;
this.operator = operator;
return this;
}
public OperatorWrapper<P, V> engine(DagEngine<?> engine) {
if (engine == null) {
throw new TaskFlowException("DagEngine is null");
}
if (instanceName == null) {
throw new TaskFlowException("id is null");
}
if (engine.getWrapperMap().containsKey(instanceName)) {
throw new TaskFlowException("id duplicates");
}
engine.getWrapperMap().put(instanceName, this);
this.engine = engine;
return this;
}
public OperatorWrapper<P, V> depend(String ... wrapperIds) {
if (wrapperIds == null) {
return this;
}
for (String wrapperId : wrapperIds) {
this.depend(wrapperId, true);
}
return this;
}
public OperatorWrapper<P, V> depend(String wrapperId, boolean isMust) {
if (wrapperId == null) {
return this;
}
if (dependWrapperIdMap == null) {
dependWrapperIdMap = new HashMap<>();
}
dependWrapperIdMap.put(wrapperId, isMust);
return this;
}
public boolean compareAndSetState(int expect, int update) {
return this.wrapperState.compareAndSet(expect, update);
}
@Override
public String toString() {
return "OperatorWrapper{" +
"id='" + instanceName + '\'' +
", operator=" + operator +
'}';
}
public String getInstanceName() {
return instanceName;
}
public AtomicInteger getIndegree() {
return indegree;
}
public IOperator<P, V> getOperator() {
return operator;
}
public OperatorResult<V> getOperatorResult() {
return operatorResult;
}
public AtomicInteger getWrapperState() {
return wrapperState;
}
public Set<OperatorWrapper<?, ?>> getNextWrappers() {
return nextWrappers;
}
public void setNextWrappers(Set<OperatorWrapper<?, ?>> nextWrappers) {
this.nextWrappers = nextWrappers;
}
public Set<OperatorWrapper<?, ?>> getDependWrappers() {
return dependWrappers;
}
public void setDependWrappers(Set<OperatorWrapper<?, ?>> dependWrappers) {
this.dependWrappers = dependWrappers;
}
public Set<OperatorWrapper<?, ?>> getSelfIsMustSet() {
return selfIsMustSet;
}
public void setSelfIsMustSet(Set<OperatorWrapper<?, ?>> selfIsMustSet) {
this.selfIsMustSet = selfIsMustSet;
}
public void setThread(Thread thread) {
this.thread = thread;
}
public Thread getThread() {
return thread;
}
public void setEngine(DagEngine<?> engine) {
this.engine = engine;
this.engine.getWrapperMap().put(this.getInstanceName(), this);
}
public void setInstanceName(String instanceName) {
this.instanceName = instanceName;
}
public Map<String, Boolean> getDependWrapperIdMap() {
return dependWrapperIdMap;
}
public DagEngine<?> getEngine() {
return engine;
}
public OpConfig getOpConfig() {
return opConfig;
}
}
lowcode.core.framework2.OpConfig
OP类的元信息
package com.example.lowcode.core.framework2;
import com.example.lowcode.core.dto.ComponentInfo;
import lombok.Data;
import java.util.List;
import java.util.Map;
/**
* @author llxzdmd
*/
@Data
public class OpConfig {
private ComponentInfo componentInfo;
private String className;
/**
* 其它配置
*/
private Map<String, Object> extMap;
}
lowcode.core.model2.DagWrapperState
package com.example.lowcode.core.model2;
import lombok.Getter;
/**
* 节点执行状态枚举
*
* @author llxzdmd
* @version DagWrapperState.java, 2024年02月18日 15:45 llxzdmd
*/
@Getter
public enum DagWrapperState {
/**
* 初始状态
*/
INIT(0),
/**
* 执行中
*/
RUNNING(1),
/**
* 执行结束
*/
FINISH(2),
/**
* 节点执行异常
*/
ERROR(3),
/**
* 跳过当前节点
*/
SKIP(4);
private final int stage;
DagWrapperState(int stage){
this.stage = stage;
}
}
lowcode.core.model2.ResultState
package com.example.lowcode.core.model2;
/**
* 节点执行结果枚举
*
* @author llxzdmd
*/
public enum ResultState {
/**
* 成功
*/
SUCCESS,
/**
* 超时
*/
TIMEOUT,
/**
* 异常
*/
EXCEPTION,
/**
* 默认
*/
DEFAULT
}
com.example.lowcode.core.exception.FlowConfigException
自定义异常类，先随便定义了三个，暂不特殊处理什么。
package com.example.lowcode.core.exception;
/**
* @author llxzdmd
*/
public class FlowConfigException extends RuntimeException {
public FlowConfigException (String message) {
super(message);
}
public FlowConfigException (String message, Throwable cause) {
super(message, cause);
}
}
lowcode.core.exception.FlowExecutionException
package com.example.lowcode.core.exception;
/**
* @author llxzdmd
*/
public class FlowExecutionException extends RuntimeException {
public FlowExecutionException (String message) {
super(message);
}
public FlowExecutionException (String message, Throwable cause) {
super(message, cause);
}
}
lowcode.core.exception.TaskFlowException
package com.example.lowcode.core.exception;
/**
* @author llxzdmd
*/
public class TaskFlowException extends RuntimeException {
public TaskFlowException(String message) {
super(message);
}
public TaskFlowException(Exception e) {
super(e);
}
}
lowcode.util.ConvertUtil
package com.example.lowcode.util;
import com.example.lowcode.core.framework2.OperatorWrapper;
import java.util.Set;
/**
* 类型转换工具
*
* @author llxzdmd
*/
public class ConvertUtil {
@SuppressWarnings("rawtypes")
public static OperatorWrapper[] set2Array(Set<OperatorWrapper<?, ?>> wrapperSet) {
if (wrapperSet == null || wrapperSet.isEmpty()) {
return new OperatorWrapper[0];
}
return wrapperSet.toArray(new OperatorWrapper[0]);
}
}
lowcode.core.service.RunServiceImpl
这里是修改实现类代码，新建V3方法
private static final ThreadPoolExecutor THREAD_POOL_EXECUTOR = new ThreadPoolExecutor(
20,100,60, TimeUnit.MINUTES,new LinkedBlockingDeque<>(500),
new ThreadFactoryBuilder().setNameFormat("pool-%d").build(),new ThreadPoolExecutor.AbortPolicy());
@Override
public Map<String, Object> runFlowV3(long flowId, Map<String, ComponentInfo> inputParams) {
FlowSnapshot flowSnapshot = FlowSnapshotDO.selectByFlowId(flowId);
assert flowSnapshot != null;
Flow flow = JSON.parseObject(flowSnapshot.getJsonParam(), new TypeReference<>() {
});
DagEngine<Map<String, Object>> engine = new FlowEngineBuilder<Map<String, Object>>()
.setFlow(flow)
.setInputParams(inputParams)
.setExecutor(THREAD_POOL_EXECUTOR)
.buildEngine();
// 这里没有直接用.builder()构造，因为要构造Map<String, Object>泛型
// FlowEngineBuilder<Object> build = FlowEngineBuilder.builder()
// .flow(flow)
// .inputParams(inputParams)
// .executor(THREAD_POOL_EXECUTOR)
// .build();
// DagEngine<Object> engine = build.buildEngine();
engine.runAndWait();
if (engine.getEx() != null) {
throw new FlowExecutionException(String.format("【%s：%s】执行异常，原因：%s", flow.getId(), flow.getName(), engine.getEx().getMessage()), engine.getEx());
}
return engine.getOutput().getResult();
}测试
这里还用上一篇的示例进行测试，但要注意因为架构增加了组件之间的连线，因此测试方法的代码也需要加一部分，这里也贴上示例：
lowcode.service.DagRunServiceTest
package com.example.lowcode.service;
import com.alibaba.fastjson.JSON;
import com.example.lowcode.core.dto.ComponentInfo;
import com.example.lowcode.core.dto.ComponentParam;
import com.example.lowcode.core.dto.FlowNode;
import com.example.lowcode.core.dto2.Edge;
import com.example.lowcode.core.dto2.Flow;
import com.example.lowcode.core.model.ComponentTypeEnum;
import com.example.lowcode.core.model.ParamTypeEnum;
import com.example.lowcode.core.service.RunService;
import com.example.lowcode.dao.mock.ComponentMetaInfoDO;
import com.example.lowcode.dao.mock.FlowSnapshotDO;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* @author llxzdmd
* @version DagRunServiceTest.java, 2024年02月19日 14:41 llxzdmd
*/
@SpringBootTest
public class DagRunServiceTest {
@Autowired
private RunService runService;
@Test
public void runFlowV3Test() {
saveFlowSnapshotTest();
Map<String, Object> result = runService.runFlowV3(3L, new HashMap<>());
System.out.println(result);
}
@Test
public void saveFlowSnapshotTest() {
Flow flow = new Flow();
flow.setId(3L);
flow.setName("测试流2.0");
List<FlowNode> flowNodeList = new ArrayList<>();
List<Edge> edgeList = new ArrayList<>();
flow.setNodeInstances(flowNodeList);
flow.setEdgeInstances(edgeList);
// 1.组件1
FlowNode flowNode1 = new FlowNode();
flowNode1.setNodeName("去重组件(前端展示的组件名)");
// flowNode1.setOrder(2);
flowNode1.setComponentId(2L);
flowNode1.setComponentName("DistinctFilter");
flowNode1.setType(ComponentTypeEnum.FILTER);
// 去重过滤器填参数
ComponentInfo componentInfo1 = new ComponentInfo();
componentInfo1.setComponentId(2L);
componentInfo1.setFlowId(3L);
Map<String, ComponentParam> inputMap1 = new HashMap<>();
inputMap1.put("list", buildComponentParam(ParamTypeEnum.LIST, "list123", "需要去重的集合", "${inputList}", true));
Map<String, ComponentParam> outputMap1 = new HashMap<>();
outputMap1.put("result", buildComponentParam(ParamTypeEnum.STRING, "result", "去重组件执行结果", "result2", true));
componentInfo1.setInputs(inputMap1);
componentInfo1.setOutputs(outputMap1);
flowNode1.setComponentInfo(componentInfo1);
flowNode1.setComponentMetaInfo(ComponentMetaInfoDO.mockData().get(1));
// 2.组件2
FlowNode flowNode2 = new FlowNode();
flowNode2.setNodeName("分页组件(前端展示的组件名)");
// flowNode2.setOrder(3);
flowNode2.setComponentId(4L);
flowNode2.setComponentName("PageFilter");
flowNode2.setType(ComponentTypeEnum.FILTER);
// 分页过滤器填参数
ComponentInfo componentInfo2 = new ComponentInfo();
componentInfo2.setComponentId(4L);
componentInfo2.setFlowId(3L);
Map<String, ComponentParam> inputMap2 = new HashMap<>();
inputMap2.put("list", buildComponentParam(ParamTypeEnum.LIST, "list456", "需要分页的集合", "${result2}", true));
inputMap2.put("pageNum", buildComponentParam(ParamTypeEnum.INT, "pageNum111", "起始在第几页，默认1", 2, false));
inputMap2.put("pageSize", buildComponentParam(ParamTypeEnum.INT, "pageSize111", "分页大小，默认10", 3, false));
Map<String, ComponentParam> outputMap2 = new HashMap<>();
outputMap2.put("result", buildComponentParam(ParamTypeEnum.STRING, "result", "分页组件执行结果", "result3", true));
componentInfo2.setInputs(inputMap2);
componentInfo2.setOutputs(outputMap2);
flowNode2.setComponentInfo(componentInfo2);
flowNode2.setComponentMetaInfo(ComponentMetaInfoDO.mockData().get(3));
// 3.组件3
FlowNode flowNode3 = new FlowNode();
flowNode3.setNodeName("inputParam");
// flowNode3.setOrder(1);
flowNode3.setComponentId(1L);
flowNode3.setComponentName("BuildParamComponent");
flowNode3.setType(ComponentTypeEnum.FILTER);
// 填参数
ComponentInfo componentInfo3 = new ComponentInfo();
componentInfo3.setComponentId(1L);
componentInfo3.setFlowId(3L);
List<String> stringList = new ArrayList<>();
stringList.add("a");
stringList.add("b");
stringList.add("c");
stringList.add("d");
stringList.add("e");
stringList.add("f");
stringList.add("g");
stringList.add("c");
stringList.add("a");
Map<String, ComponentParam> inputMap3 = new HashMap<>();
inputMap3.put("params", buildComponentParam(ParamTypeEnum.MAP, "list", "入口入参list", new HashMap<>() {{
put("inputList", stringList);
}}, true));
Map<String, ComponentParam> outputMap3 = new HashMap<>();
outputMap3.put("result", buildComponentParam(ParamTypeEnum.STRING, "result", "入参结果", "result1", true));
componentInfo3.setInputs(inputMap3);
componentInfo3.setOutputs(outputMap3);
flowNode3.setComponentInfo(componentInfo3);
flowNode3.setComponentMetaInfo(ComponentMetaInfoDO.mockData().get(0));
// 4.组件4
FlowNode flowNode4 = new FlowNode();
flowNode4.setNodeName("HttpClient-123");
// flowNode4.setOrder(4);
flowNode4.setComponentId(3L);
flowNode4.setComponentName("HttpClient");
flowNode4.setType(ComponentTypeEnum.FILTER);
// 填参数
ComponentInfo componentInfo4 = new ComponentInfo();
componentInfo4.setComponentId(3L);
componentInfo4.setFlowId(3L);
componentInfo4.setInputs(new HashMap<>() {{
put("url", new ComponentParam().setType(ParamTypeEnum.STRING).setRequired(true).setName("url").setValue("https://www.baidu.com"));
put("method", new ComponentParam().setType(ParamTypeEnum.STRING).setRequired(true).setName("method").setValue("get"));
put("params", new ComponentParam().setType(ParamTypeEnum.STRING).setRequired(false).setName("params").setValue(new HashMap<>()));
put("headers", new ComponentParam().setType(ParamTypeEnum.STRING).setRequired(false).setName("headers").setValue(new HashMap<>()));
}});
componentInfo4.setOutputs(new HashMap<>() {{
put("result", new ComponentParam().setType(ParamTypeEnum.STRING).setRequired(true).setName("result").setValue("result4"));
}});
flowNode4.setComponentInfo(componentInfo4);
flowNode4.setComponentMetaInfo(ComponentMetaInfoDO.mockData().get(2));
// 5.组件5
FlowNode flowNode5 = new FlowNode();
flowNode5.setNodeName("outputParam");
// flowNode5.setOrder(5);
flowNode5.setComponentId(1L);
flowNode5.setComponentName("BuildParamComponent");
flowNode5.setType(ComponentTypeEnum.FILTER);
// 填参数
ComponentInfo componentInfo5 = new ComponentInfo();
componentInfo5.setComponentId(1L);
componentInfo5.setFlowId(3L);
Map<String, ComponentParam> inputMap5 = new HashMap<>();
inputMap5.put("params", buildComponentParam(ParamTypeEnum.MAP, "params", "呃呃", new HashMap<>() {{
put("pageResult", "${result3}");
put("httpResult", "${result4}");
}}, true));
Map<String, ComponentParam> outputMap5 = new HashMap<>();
outputMap5.put("result", buildComponentParam(ParamTypeEnum.STRING, "result", "输出结果", "result5", true));
componentInfo5.setInputs(inputMap5);
componentInfo5.setOutputs(outputMap5);
flowNode5.setComponentInfo(componentInfo5);
flowNode5.setComponentMetaInfo(ComponentMetaInfoDO.mockData().get(0));
flowNodeList.add(flowNode1);
flowNodeList.add(flowNode2);
flowNodeList.add(flowNode3);
flowNodeList.add(flowNode4);
flowNodeList.add(flowNode5);
Edge edge1 = new Edge();
edge1.setSourceName("inputParam");
edge1.setTargetName("去重组件(前端展示的组件名)");
Edge edge2 = new Edge();
edge2.setSourceName("去重组件(前端展示的组件名)");
edge2.setTargetName("分页组件(前端展示的组件名)");
Edge edge3 = new Edge();
edge3.setSourceName("inputParam");
edge3.setTargetName("HttpClient-123");
Edge edge4 = new Edge();
edge4.setSourceName("分页组件(前端展示的组件名)");
edge4.setTargetName("outputParam");
Edge edge5 = new Edge();
edge5.setSourceName("HttpClient-123");
edge5.setTargetName("outputParam");
edgeList.add(edge1);
edgeList.add(edge2);
edgeList.add(edge3);
edgeList.add(edge4);
edgeList.add(edge5);
// 调用保存组件流快照的接口，模拟入库
FlowSnapshotDO.insertData(3L, JSON.toJSONString(flow));
}
private ComponentParam buildComponentParam(ParamTypeEnum typeEnum, String name, String desc, Object value, boolean required) {
ComponentParam componentParam = new ComponentParam();
componentParam.setType(typeEnum);
componentParam.setName(name);
componentParam.setDesc(desc);
componentParam.setValue(value);
componentParam.setRequired(required);
return componentParam;
}
}
另外为了方便演示，这里还在每个组件前面展示了当前线程名；
以及纠正一个之前的错误，本次v2.0升级架构后，不需要在每个组件最后手动把结果放入上下文了，放到执行的方法中就可以了，看来还是设计模式学的不到位啊。
最后注：这次把ComponentContext类中的上下文定义为ConcurrentHashMap了，防止多个组件同时修改上下文时出现线程安全问题。
运行结果：
可以看到用到了两个线程，有兴趣的话可以多试试其他的测试用例，等有时间我也会补充上。
这里因为好奇线程安全问题，测试了一下并发场景
private static final ThreadPoolExecutor THREAD_POOL_EXECUTOR = new ThreadPoolExecutor(
20, 1000, 60, TimeUnit.MINUTES, new LinkedBlockingDeque<>(500),
new ThreadFactoryBuilder().setNameFormat("pool-%d").build(), new ThreadPoolExecutor.AbortPolicy());
@Test
public void testJUC1() throws InterruptedException {
CountDownLatch latch = new CountDownLatch(1000);
int numberOfTasks = 1000;
try {
saveFlowSnapshotTest();
} catch (Exception e) {
System.out.println("ERROR1");
System.out.println(e.getMessage());
}
for (int i = 0; i < numberOfTasks; i++) {
THREAD_POOL_EXECUTOR.submit(() -> {
try {
Map<String, Object> result = runService.runFlowV3(3L, new HashMap<>());
} catch (Exception e) {
System.out.println("ERROR2");
System.out.println(e.getMessage());
}
try {
latch.countDown();
} catch (Exception e) {
System.out.println("ERROR3");
System.out.println(e.getMessage());
}
});
}
latch.await();
}发现每次运行都会报几次 ERROR2 错误
显示原因是DagEngine类的output属性获取为null，在几个关键的位置都设置埋点后定位到是在runFlowV3方法的最后一行：“return engine.getOutput().getResult();”处报的这个错。
自己研究了一大会儿，也和一个同事讨论了半天，都想不出为什么会有这个问题。
带着疑惑又去跑了github上的源码，发现尽管源码里面包了ThreadLocal等结构，也会报这个错。所以这个问题暂时无解了，不知到会不会有大佬看到这个问题并解答一下。
总结
先不写了之后补充
扫一扫
2832
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
