flink 参数之 -yn and -p

最新推荐文章于 2025-09-11 13:16:53 发布

原创最新推荐文章于 2025-09-11 13:16:53 发布 · 2.3k 阅读

CC 4.0 BY-SA版权

本文介绍了 Apache Flink 中并行度设置的过程及原理。当用户通过 `-p` 参数指定并行度时，系统将根据集群可用的 YARN 容器槽位数调整每个 TaskManager 的槽位分配，确保并行任务可以顺利运行。

-yn container , also known as task manager count.

-p, parallelism , it is the count the job be divided into. the count of subtask.

please see following code:

when set the -p, whatever, flink run will popup a message. "but the ...."

// convenience
	int userParallelism = Integer.valueOf(cmd.getOptionValue(CliFrontendParser.PARALLELISM_OPTION.getOpt(),"-1"));
	int maxSlots = slotsPerTaskManager * numberTaskManagers;
	if (userParallelism != -1) {
	int slotsPerTM = (int) Math.ceil((double) userParallelism/ numberTaskManagers);
	String message = "The YARN cluster has" + maxSlots + " slots available, " +
	"but the user requested a parallelism of" + userParallelism +" on YARN." +
	"Each of the" + numberTaskManagers +" TaskManagers" +
	"will get" + slotsPerTM + " slots.";
	logAndSysout(message);
	slotsPerTaskManager = slotsPerTM;
	}