azkaban集群多节点模式配置

最新推荐文章于 2025-06-30 11:24:39 发布

_delin

最新推荐文章于 2025-06-30 11:24:39 发布

阅读量9.3k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： hadoop

本文链接：https://blog.youkuaiyun.com/jacklin929/article/details/53742638

hadoop 专栏收录该内容

12 篇文章

订阅专栏

本文介绍如何配置Azkaban以支持多节点执行任务。主要内容包括修改AzkabanWebServer配置文件azkaban.properties启用多节点模式，并设置执行服务器的选择策略。此外，还需要在MySQL数据库中添加各执行服务器的信息。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

配置多节点执行服务器的时候，需要在AzkabanWebServer的配置文件azkaban.properties里添加

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

以确认使用的是分布式方式，随后提交的job会根据情况自行选择执行服务器，否则默认只使用本地执行服务器，切记！

之后需要在mysql的azkaban库中添加各个执行服务器的ip/域名和端口：

insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);

至此，azkaban的分布式配置才算成功。

配置多节点执行服务器的时候，需要在AzkabanWebServer的配置文件azkaban.properties里添加

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

以确认使用的是分布式方式，随后提交的job会根据情况自行选择执行服务器，否则默认只使用本地执行服务器，切记！

之后需要在mysql的azkaban库中添加各个执行服务器的ip/域名和端口：

insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);

至此，azkaban的分布式配置才算成功。

以下来源官网文档： http://azkaban.github.io/azkaban/docs/latest/

Configuring AzabanWebServer and AzkabanExecutorServer clients

The Executor server needs to be setup with a port, and the AzabanWebServer will need to know what this port is.

The following properties need to be set on AzkabanExecutorServer's azkaban.properties.

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30

Single Executor Mode

By default the executor.port is set to 12321. The AzkabanWebServer will have to point to this port as well.

This is done by setting the following property in AzkabanWebServer's azkaban.properties.

executor.port=12321

Multiple Executor Mode

We need to enable multiple executor mode in webserver configuration if we want to run in multiple executor mode. Please verify that you have following properties in azkaban.properties. azkaban.use.multiple.executors and azkaban.executorselector.comparator.* are required properties. Please note that azkaban.use.multiple.executors is not honoured in multiple exeutor mode.

azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

These changes are only picked up after restarting the servers.

Configure executors for Multiple Executor Mode

We don't have a executor management UI at this point of Time. Executors need to be configured in database. For examples:-

Insert all executors into the mysql DB for executor setups. Verify that correct executors are active in executors table.insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);