Flink源码篇 No.8-任务提交之启动TaskManager（per-job on yarn）

第1章 注册回调

在上一篇文章中，启动nodeManagerClient的时候，注册了一个CallbackHandler回调yarnContainerEventHandler

org.apache.flink.yarn.YarnResourceManagerDriver#initializeInternal

@Override
protected void initializeInternal() throws Exception {
	final YarnContainerEventHandler yarnContainerEventHandler = new YarnContainerEventHandler();
	try {
		// TODO 创建并启动Yarn的resourceManager客户端，用于向Yarn申请资源
		resourceManagerClient = yarnResourceManagerClientFactory.createResourceManagerClient(
			yarnHeartbeatIntervalMillis,
			yarnContainerEventHandler);
		resourceManagerClient.init(yarnConfig);
		resourceManagerClient.start();

		// TODO 注册AppMaster
		final RegisterApplicationMasterResponse registerApplicationMasterResponse = registerApplicationMaster();
		getContainersFromPreviousAttempts(registerApplicationMasterResponse);
		taskExecutorProcessSpecContainerResourcePriorityAdapter =
			new TaskExecutorProcessSpecContainerResourcePriorityAdapter(
				registerApplicationMasterResponse.getMaximumResourceCapability(),
				ExternalResourceUtils.getExternalResources(flinkConfig, YarnConfigOptions.EXTERNAL_RESOURCE_YARN_CONFIG_KEY_SUFFIX));
	} catch (Exception e) {
		throw new ResourceManagerException("Could not start resource manager client.", e);
	}

	// TODO 创建并启动Yarn的nodeManager课户端，用于启动taskManager
	nodeManagerClient = yarnNodeManagerClientFactory.createNodeManagerClient(yarnContainerEventHandler);
	nodeManagerClient.init(yarnConfig);
	nodeManagerClient.start();
}

第2章 注册TaskManager启动类

在yarnContainerEventHandler中，有一个容器分配的回调方法

org.apache.flink.yarn.YarnResourceManagerDriver.YarnContainerEventHandler#onContainersAllocated

@Override
public void onContainersAllocated(List<Container> containers) {
	runAsyncWithFatalHandler(() -> {
		checkInitialized();
		log.info("Received {} containers.", containers.size());

		for (Map.Entry<Priority, List<Container>> entry : groupContainerByPriority(containers).entrySet()) {
			// TODO 按优先级分配容器
			onContainersOfPriorityAllocated(entry.getKey(), entry.getValue());
		}

		// if we are waiting for no further containers, we can go to the
		// regular heartbeat interval
		if (getNumRequestedNotAllocatedWorkers() <= 0) {
			resourceManagerClient.setHeartbeatInterval(yarnHeartbeatIntervalMillis);
		}
	});
}

org.apache.flink.yarn.YarnResourceManagerDriver#onContainersOfPriorityAllocated

private void onContainersOfPriorityAllocated(Priority priority, List<Container> containers) {
	//...
	
	int numAccepted = 0;
	while (containerIterator.hasNext() && pendingContainerRequestIterator.hasNext()) {
		final Container container = containerIterator.next();
		final AMRMClient.ContainerRequest pendingRequest = pendingContainerRequestIterator.next();
		final ResourceID resourceId = getContainerResourceId(container);

		final CompletableFuture<YarnWorkerNode> requestResourceFuture = pendingRequestResourceFutures.poll();
		Preconditions.checkState(requestResourceFuture != null);

		if (pendingRequestResourceFutures.isEmpty()) {
			requestResourceFutures.remove(taskExecutorProcessSpec);
		}

		// TODO 在容器中启动TaskExecutor，异步启动
		startTaskExecutorInContainerAsync(container, taskExecutorProcessSpec, resourceId, requestResourceFuture);
		removeContainerRequest(pendingRequest);

		numAccepted++;
	}

	// ...
}

org.apache.flink.yarn.YarnResourceManagerDriver#startTaskExecutorInContainerAsync

private void startTaskExecutorInContainerAsync(
		Container container,
		TaskExecutorProcessSpec taskExecutorProcessSpec,
		ResourceID resourceId,
		CompletableFuture<YarnWorkerNode> requestResourceFuture) {
	// TODO 创建TaskExecutor
	final CompletableFuture<ContainerLaunchContext> containerLaunchContextFuture =
		FutureUtils.supplyAsync(() -> createTaskExecutorLaunchContext(
			resourceId, container.getNodeId().getHost(), taskExecutorProcessSpec), getIoExecutor());

	FutureUtils.assertNoException(
		containerLaunchContextFuture.handleAsync((context, exception) -> {
			if (exception == null) {
				nodeManagerClient.startContainerAsync(container, context);
				requestResourceFuture.complete(new YarnWorkerNode(container, resourceId));
			} else {
				requestResourceFuture.completeExceptionally(exception);
			}
			return null;
		}, getMainThreadExecutor()));
}

org.apache.flink.yarn.YarnResourceManagerDriver#createTaskExecutorLaunchContext

private ContainerLaunchContext createTaskExecutorLaunchContext(
	ResourceID containerId,
	String host,
	TaskExecutorProcessSpec taskExecutorProcessSpec) throws Exception {

	// init the ContainerLaunchContext
	final String currDir = configuration.getCurrentDir();

	final ContaineredTaskManagerParameters taskManagerParameters =
		ContaineredTaskManagerParameters.create(flinkConfig, taskExecutorProcessSpec);

	log.info("TaskExecutor {} will be started on {} with {}.",
		containerId.getStringWithMetadata(),
		host,
		taskExecutorProcessSpec);

	final Configuration taskManagerConfig = BootstrapTools.cloneConfiguration(flinkConfig);
	taskManagerConfig.set(TaskManagerOptions.TASK_MANAGER_RESOURCE_ID, containerId.getResourceIdString());
	taskManagerConfig.set(TaskManagerOptionsInternal.TASK_MANAGER_RESOURCE_ID_METADATA, containerId.getMetadata());

	final String taskManagerDynamicProperties =
		BootstrapTools.getDynamicPropertiesAsString(flinkClientConfig, taskManagerConfig);

	log.debug("TaskManager configuration: {}", taskManagerConfig);

	// TODO 创建TaskExecutor
	// TODO 注册taskManagerMainClass，容器内调用这个YarnTaskExecutorRunner.class的main方法来启动Flink的Taskmanager
	final ContainerLaunchContext taskExecutorLaunchContext = Utils.createTaskExecutorContext(
		flinkConfig,
		yarnConfig,
		configuration,
		taskManagerParameters,
		taskManagerDynamicProperties,
		currDir,
		YarnTaskExecutorRunner.class,
		log);

	taskExecutorLaunchContext.getEnvironment()
		.put(ENV_FLINK_NODE_ID, host);
	return taskExecutorLaunchContext;
}

当TaskExecutor启动时，就会调用YarnTaskExecutorRunner的main方法来启动TaskManager。

第3章 启动TaskManager

 YarnTaskExecutorRunner就是Yarn模式下的TaskManager启动类，我们接着往下看：

org.apache.flink.yarn.YarnTaskExecutorRunner#main

/**
 * The entry point for the YARN task executor runner.
 *
 * @param args The command line arguments.
 */
public static void main(String[] args) {
	EnvironmentInformation.logEnvironmentInfo(LOG, "YARN TaskExecutor runner", args);
	SignalHandler.register(LOG);
	JvmShutdownSafeguard.installAsShutdownHook(LOG);

	// TODO 运行TaskManager
	runTaskManagerSecurely(args);
}

org.apache.flink.yarn.YarnTaskExecutorRunner#runTaskManagerSecurely

private static void runTaskManagerSecurely(String[] args) {
	try {
		LOG.debug("All environment variables: {}", ENV);

		// 获得环境当前路径
		final String currDir = ENV.get(Environment.PWD.key());
		LOG.info("Current working Directory: {}", currDir);

		// 加载配置
		final Configuration configuration = TaskManagerRunner.loadConfiguration(args);
		setupAndModifyConfiguration(configuration, currDir, ENV);

		// TODO 运行TaskManager
		TaskManagerRunner.runTaskManagerSecurely(configuration);
	}
	catch (Throwable t) {
		final Throwable strippedThrowable = ExceptionUtils.stripException(t, UndeclaredThrowableException.class);
		// make sure that everything whatever ends up in the log
		LOG.error("YARN TaskManager initialization failed.", strippedThrowable);
		System.exit(INIT_ERROR_EXIT_CODE);
	}
}

org.apache.flink.runtime.taskexecutor.TaskManagerRunner#runTaskManagerSecurely(org.apache.flink.configuration.Configuration)

public static void runTaskManagerSecurely(Configuration configuration) throws Exception {
	replaceGracefulExitWithHaltIfConfigured(configuration);
	final PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(configuration);
	// 文件系统初始化
	FileSystem.initialize(configuration, pluginManager);

	// 安装配置信息
	SecurityUtils.install(new SecurityConfiguration(configuration));

	// TODO 运行TaskManager
	SecurityUtils.getInstalledContext().runSecured(() -> {
		runTaskManager(configuration, pluginManager);
		return null;
	});
}

org.apache.flink.runtime.taskexecutor.TaskManagerRunner#runTaskManager

public static void runTaskManager(Configuration configuration, PluginManager pluginManager) throws Exception {
	final TaskManagerRunner taskManagerRunner = new TaskManagerRunner(configuration, pluginManager, TaskManagerRunner::createTaskExecutorService);

	// TODO 启动TaskManager
	taskManagerRunner.start();
}

org.apache.flink.runtime.taskexecutor.TaskManagerRunner#start

public void start() throws Exception {
	taskExecutorService.start();
}

org.apache.flink.runtime.taskexecutor.TaskManagerRunner.TaskExecutorService#start

实现类是：org.apache.flink.runtime.taskexecutor.TaskExecutorToServiceAdapter

org.apache.flink.runtime.taskexecutor.TaskExecutorToServiceAdapter#start

@Override
public void start() {
	// TODO RPC的方式启动TaskExecutor => TaskExecutor的onStart方法
	taskExecutor.start();
}

org.apache.flink.runtime.taskexecutor.TaskExecutor#onStart

@Override
public void onStart() throws Exception {
	try {
		// TODO 启动TaskExecutor服务
		startTaskExecutorServices();
	} catch (Throwable t) {
		final TaskManagerException exception = new TaskManagerException(String.format("Could not start the TaskExecutor %s", getAddress()), t);
		onFatalError(exception);
		throw exception;
	}

	startRegistrationTimeout();
}

到这里，TaskManager就被Yarn启动起来，而TaskManager启动后，其内部又做了什么呢？下篇文章咱们继续。