azkaban web 端添加配置 azkaban.properties:
azkaban web 端添加配置 azkaban.properties
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
azkaban.activeexecutor.refresh.milisecinterval=1000
azkaban.activeexecutor.refresh.flowinterval=1
azkaban.executorinfo.refresh.maxThreads=1
Servers Executor servers:
192.168.11.164
192.168.11.142
192.168.11.105
Azkaban web:
192.168.11.105
Mysql:
192.168.11.86
use azkaban;
NO. Operation Tables & SQL Result(my test result sample)
1 After R&D config the azkaban web/executors and restart,executors should be automatically registered successfully; Check the registration results:
select * from executors;
| id | host | port | active |
+----+-----------------------------------+-------+--------+
| 62 | THA-168-11-164.h.chinabank.com.cn | 12321 | 1 |
| 64 | THA-168-11-142.h.chinabank.com.cn | 12321 | 1 |
| 66 | THA-168-11-105.h.chinabank.com.cn | 12321 | 1 |
2 Execute the old Azkaban job, its should be randomly assigned to one of the executors ;
Check this job's executor id:
SELECT executor_id FROM `execution_flows` where exec_id= executor_id
+-------------+
| 64 |
3 Create data ingestion job,it should be randomly assigned to one of the executors ; Check the executor id :
SELECT executor_id FROM `execution_flows` where exec_id=
| executor_id |
+-------------+
| 66 |
4 Execute multiple Azkaban jobs ,they assigned to the executor randomly; Check the executors id:
SELECT executor_id FROM `execution_flows` where exec_id=;
| executor_id |
+-------------+
| 64 |
| 66 |
| 64 |
| 66 |
| 64 |
| 62 |
| 64 |
5 The same Azkaban job execute on the different executor servers(Run the job many times manually); Check the executors id:
SELECT project_id, executor_id FROM `execution_flows` ORDER BY exec_id desc LIMIT 2;
| project_id | executor_id |
+------------+-------------+
| 2562 | 66 |
| 2562 | 64 |
6 When Azkaban job is running, stop the server it is using,this job's execution result should be failed, re-executed it manually should be run on another executor: Check the executors id:
SELECT project_id, exec_id,executor_id FROM `execution_flows` ORDER BY exec_id desc limit 5;
| project_id | executor_id |
+------------+-------------+
| 2581 | 64 |
| 2581 | 66 |
7 Multiple Azkaban jobs are executed at the same time,they should be randomly assigned to the executor; Check the executors id:
SELECT project_id, exec_id,executor_id FROM `execution_flows` ORDER BY exec_id desc limit 55;
(Since we reconfigured 105 server, its id changed from 66 to 68)
| project_id | exec_id | executor_id |
+------------+---------+-------------+
| 2624 | 88642 | 62 |
| 2627 | 88641 | 62 |
| 2619 | 88640 | 62 |
| 2353 | 88639 | 64 |