问题:
java.lang.IllegalArgumentException: Source: 21 is not a parallel source
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Source: 21 is not a parallel source
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:80) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:122) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
at cn.com.bsfit.pipeace.mgr.job.client.flink.yarn.perjob.FlinkYarnPerJobClusterClient.submit(FlinkYarnPerJobClusterClient.java:103) ~[classes!/:2.4.0.2.0-huaXing-SNAPSHOT]
at cn.com.bsfit.pipeace.mgr.job.client.flink.yarn.perjob.FlinkYarnPerJobClusterClient.submit(FlinkYarnPerJobClusterClient.java:56) ~[classes!/:2.4.0.2.0-huaXing-SNAPSHOT]
at cn.com.bsfit.pipeace.mgr.orm.service.PaJobPublishAsyncService.asyncOnline(PaJobPublishAsyncService.java:201) ~[classes!/:2.4.0.2.0-huaXing-SNAPSHOT]
at cn.com.bsfit.pipeace.mgr.orm.service.PaJobPublishAsyncService$$FastClassBySpringCGLIB$$b8fb53a6.invoke(<generated>) ~[classes!/:2.4.0.2.0-huaXing-SNAPSHOT]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:750) ~[spring-aop-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) ~[spring-aop-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.lambda$doSubmit$3(AsyncExecutionAspectSupport.java:276) ~[spring-aop-5.1.13.RELEASE.jar!/:5.1.13.RELEASE]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.IllegalArgumentException: Source: 21 is not a parallel source
at org.apache.flink.streaming.api.datastream.DataStreamSource.setParallelism(DataStreamSource.java:55) ~[flink-streaming-java_2.11-1.9.1.jar!/:1.9.1]
at cn.com.bsfit.pipeace.PipeACEJobbc7a83b1f3424bb0a0d717ac17570d1c.main(PipeACEJobbc7a83b1f3424bb0a0d717ac17570d1c.java:41) ~[PipeACEJob-bc7a83b1f3424bb0a0d717ac17570d1c-10.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) ~[flink-clients_2.11-1.9.1.jar!/:1.9.1]
... 20 more
产生原因:
source组件继承的是RichSourceFunction类而不是RichParallelSourceFunction类
解决:
继承RichParallelSourceFunction类,但是不是所有的source组件都可以使用多个并行度的,比如数据库,用多个并行度只能重复采集数据,没有什么意义。