Ambari 监控信息的提取
2017年09月21日 15:14:29 阅读数：788更多
个人分类： HDP 平台的运用
版权声明：小兄弟，慢慢熬吧...... https://blog.youkuaiyun.com/u013850277/article/details/78052320
场景：
* 通过Ambari + HDP搭建的大数据基础管理平台，目前用户不想登录Ambari而想直接将Ambari的监控相关信息整合到其他的Web项目中。
如将下面资源使用用量相关的提取出来
- 1
- 2
- 3
- 解决方式：
- 本来是想着找到Ambari Dashboard功能的相关链接，然后通过模拟登录将这些视图提取出来，或者通过Ambari自带的权限管理设置用户只拥有查看Ambari Dashboard视力的权限。最终发现在不深入学习Ambari源码的情况下，上述两种方式实现起来太不容易了。
- 因而最终采用直接读取记录了Ambari监控信息的数据库，然后通过可视化工具再实现将监控视图呈现如来。
获取Ambari监控信息的数据库
- 具体操作步骤如下所示：
1、先切换用户
[root@hdp39 ~]# su postgres- 1
2、重新使用amabri用户进入postgresql(默认密码bigdata)
bash-4.1$ psql -U ambari
could not change directory to "/root"
Password for user ambari:
psql (8.4.20)
Type "help" for help.
ambari=> - 1
- 2
- 3
- 4
- 5
- 6
- 7
3、列举数据库
列举数据库，相当于mysql的show databases
ambari=> \l
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
--------------+-----------+----------+-------------+-------------+----------------------------
ambari | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres
: postgres=CTc/postgres
: ambari=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
ranger | rangerdba | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
ranger_audit | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres
: postgres=CTc/postgres
: ranger_logger=CTc/postgres
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres
: postgres=CTc/postgres
(6 rows)- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
4、列举表
列举表，相当于mysql的show tables(当前已经在ambari数据库)
ambari=> \dt
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+-------+----------
ambari | adminpermission | table | postgres
ambari | adminprincipal | table | postgres
ambari | adminprincipaltype | table | postgres
ambari | adminprivilege | table | postgres
ambari | adminresource | table | postgres
ambari | adminresourcetype | table | postgres
ambari | alert_current | table | postgres
ambari | alert_definition | table | postgres
ambari | alert_group | table | postgres
ambari | alert_group_target | table | postgres
ambari | alert_grouping | table | postgres
ambari | alert_history | table | postgres
ambari | alert_notice | table | postgres
ambari | alert_target | table | postgres
ambari | alert_target_states | table | postgres
ambari | ambari_operation_history | table | postgres
ambari | ambari_sequences | table | postgres
ambari | artifact | table | postgres
ambari | blueprint | table | postgres
ambari | blueprint_configuration | table | postgres
ambari | blueprint_setting | table | postgres
ambari | cluster_version | table | postgres
ambari | clusterconfig | table | postgres
ambari | clusterconfigmapping | table | postgres
ambari | clusterhostmapping | table | postgres
ambari | clusters | table | postgres
ambari | clusterservices | table | postgres
ambari | clusterstate | table | postgres
ambari | confgroupclusterconfigmapping | table | postgres
ambari | configgroup | table | postgres
ambari | configgrouphostmapping | table | postgres
ambari | ds_fileresourceitem_1 | table | ambari
ambari | ds_jobimpl_4 | table | ambari
ambari | ds_savedquery_5 | table | ambari
ambari | ds_testbean_2 | table | ambari
ambari | ds_udf_3 | table | ambari
ambari | execution_command | table | postgres
ambari | extension | table | postgres
ambari | extensionlink | table | postgres
ambari | groups | table | postgres
ambari | host_role_command | table | postgres
ambari | host_version | table | postgres
ambari | hostcomponentdesiredstate | table | postgres
ambari | hostcomponentstate | table | postgres
ambari | hostconfigmapping | table | postgres
ambari | hostgroup | table | postgres
ambari | hostgroup_component | table | postgres
ambari | hostgroup_configuration | table | postgres
ambari | hosts | table | postgres
ambari | hoststate | table | postgres
ambari | kerberos_descriptor | table | postgres
ambari | kerberos_principal | table | postgres
ambari | kerberos_principal_host | table | postgres
ambari | key_value_store | table | postgres
ambari | members | table | postgres
ambari | metainfo | table | postgres
ambari | permission_roleauthorization | table | postgres
ambari | qrtz_blob_triggers | table | postgres
ambari | qrtz_calendars | table | postgres
ambari | qrtz_cron_triggers | table | postgres
ambari | qrtz_fired_triggers | table | postgres
ambari | qrtz_job_details | table | postgres
ambari | qrtz_locks | table | postgres
ambari | qrtz_paused_trigger_grps | table | postgres
ambari | qrtz_scheduler_state | table | postgres
ambari | qrtz_simple_triggers | table | postgres
ambari | qrtz_simprop_triggers | table | postgres
ambari | qrtz_triggers | table | postgres
ambari | remoteambaricluster | table | postgres
ambari | remoteambariclusterservice | table | postgres
ambari | repo_version | table | postgres
ambari | request | table | postgres
ambari | requestoperationlevel | table | postgres
ambari | requestresourcefilter | table | postgres
ambari | requestschedule | table | postgres
ambari | requestschedulebatchrequest | table | postgres
ambari | role_success_criteria | table | postgres
ambari | roleauthorization | table | postgres
ambari | servicecomponent_history | table | postgres
ambari | servicecomponentdesiredstate | table | postgres
ambari | serviceconfig | table | postgres
ambari | serviceconfighosts | table | postgres
ambari | serviceconfigmapping | table | postgres
ambari | servicedesiredstate | table | postgres
ambari | setting | table | postgres
ambari | stack | table | postgres
ambari | stage | table | postgres
ambari | topology_host_info | table | postgres
ambari | topology_host_request | table | postgres
ambari | topology_host_task | table | postgres
ambari | topology_hostgroup | table | postgres
ambari | topology_logical_request | table | postgres
ambari | topology_logical_task | table | postgres
ambari | topology_request | table | postgres
ambari | upgrade | table | postgres
ambari | upgrade_group | table | postgres
ambari | upgrade_item | table | postgres
ambari | users | table | postgres
ambari | viewentity | table | postgres
ambari | viewinstance | table | postgres
ambari | viewinstancedata | table | postgres
ambari | viewinstanceproperty | table | postgres
ambari | viewmain | table | postgres
ambari | viewparameter | table | postgres
ambari | viewresource | table | postgres
ambari | viewurl | table | postgres
ambari | widget | table | postgres
ambari | widget_layout | table | postgres
ambari | widget_layout_user_widget | table | postgres
(109 rows)- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
5、查看表结构
相当于desc tblname
ambari-> \d users
Table "ambari.users"
Column | Type | Modifiers
-----------------------+-----------------------------+---------------------------------------------
user_id | integer | not null
principal_id | bigint | not null
ldap_user | integer | not null default 0
user_name | character varying(255) | not null
user_type | character varying(255) | not null default 'LOCAL'::character varying
create_time | timestamp without time zone | default now()
user_password | character varying(255) |
active | integer | not null default 1
active_widget_layouts | character varying(1024) | default NULL::character varying
Indexes:
"pk_users" PRIMARY KEY, btree (user_id)
"unq_users_0" UNIQUE, btree (user_name, user_type)
Foreign-key constraints:
"fk_users_principal_id" FOREIGN KEY (principal_id) REFERENCES adminprincipal(principal_id)
Referenced by:
TABLE "members" CONSTRAINT "fk_members_user_id" FOREIGN KEY (user_id) REFERENCES users(user_id)- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 在列出所有表时，通过表名可以大概确定 alert_current，alert_definition表，笔者第一次就直接查询alert_current 表，经过对比其最新时间戳（去掉后三位）发现记录的信息确实是当前整个平台最新记录的信息。如下：
在linux中对时间戳进行转换（去掉后三位）
[root@hdp39 ~]# date -d @1505959825
Thu Sep 21 10:10:25 CST 2017- 1
- 2
- 3
select alert_id,definition_id,latest_timestamp,latest_text,occurrences,firmness from alert_current where definition_id=24;
66 | 24 | 1505959825614 | Remaining Capacity:[151043118841], Total Capacity:[18% Used, 184952968704] - 1
- 2
- 3
在Ambari界面中发现alert_definition表的信息则是如下信息 
具体监控信息
ambari=> select definition_id from alert_definition where label='DataNode Heap Usage';
definition_id
---------------
15
ambari=> select alert_id,definition_id,latest_timestamp,latest_text,occurrences,firmness from alert_current where definition_id=15;
alert_id | definition_id | latest_timestamp | latest_text | occurrences | firmness
----------+---------------+------------------+----------------------------------------------------+-------------+----------
59 | 15 | 1505977580226 | Used Heap:[5%, 47.36313 MB], Max Heap: 1004.0 MB | 12217 | HARD
65 | 15 | 1505977585580 | Used Heap:[6%, 64.779465 MB], Max Heap: 1004.0 MB | 12218 | HARD
62 | 15 | 1505977581769 | Used Heap:[13%, 130.01476 MB], Max Heap: 1004.0 MB | 12218 | HARD
(3 rows)- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
通过Ambari接口获取相关信息
笔者比较懒并没有认真研究Ambari 接口，因而直接通过Debug模式查看其调用的接口，然后模拟登录并发送对应的接口获取相关信息。如下，获取集群CPU使用情况：
通过上图可以获知，发送对应的请求可获得Json相关信息。
- 最终可通过BI工具或Echarts等工具对相关信息进行可视化呈现，如下只是简单说明可以这么做而已。
由上述可知，通过alert_definition 与alert_current 表可得出具体最新的监控信息，当然还可以通过组合其他表而得出更多的监控信息。
277
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
