CDH使用秘籍（一）：Cloudera Manager和Managed Service的数据库

文章来源：http://blog.youkuaiyun.com/shifenglov/article/details/41115001

 

CDH使用秘籍（一文章来源）：Cloudera Manager和Managed Service的数据库

分类： Hadoop 2014-11-14 10:51  1130人阅读  评论(0)  收藏  举报

CDH Cloudera Manager Managed Service Host Monitor Service Monitor

背景

从业务发展需求，大数据平台需要使用spark作为机器学习、数据挖掘、实时计算等工作，所以决定使用Cloudera Manager5.2.0版本和CDH5。

以前搭建过Cloudera Manager4.8.2和CDH4，在搭建Cloudera Manager5.2.0版本的时候，发现相应的Service Host Monitor 和 Service Monitor不能配置外部表，刚开是还以为是配置出错，后来才发现应该是新版本的Cloudera的存储改变方式了。查了很多文档，果然发现，新版本中 Service Host Monitor 和 ServicMonitor e 不需要配置数据库，默认使用内置存储方式，并且不能修改。

概述

Cloudera Manager uses databases to store information about the Cloudera Manager configuration, as well as information such as the health of the system or task progress. For quick, simple installations, Cloudera Manager can install and configure an embedded PostgreSQL database as part of the Cloudera Manager installation process. In addition, some CDH services use databases and are automatically configured to use a default database. If you plan to use the embedded and default databases provided during the Cloudera Manager installation, see Installation Path A - Automated Installation by Cloudera Manager.

Although the embedded database is useful for getting started quickly, you can also use your own  PostgreSQL, MySQL, or Oracle database for the Cloudera Manager Server and services that use databases.

需要的数据库

    

The  Cloudera Manager Server, Activity Monitor, Reports Manager, Hive Metastore, Sentry Server, Cloudera Navigator Audit Server , and  Cloudera Navigator Metadata Server  all require databases. The type of data contained in the databases and their estimated sizes are as follows:

• Cloudera Manager - Contains all the information about services you have configured and their role assignments, all configuration history, commands, users, and running processes. This relatively small database (<100 MB) is the most important to back up.
• Activity Monitor - Contains information about past activities. In large clusters, this database can grow large. Configuring an Activity Monitor database is only necessary if a MapReduce service is deployed.
• Reports Manager - Tracks disk utilization and processing activities over time. Medium-sized.
• Hive Metastore - Contains Hive metadata. Relatively small.
• Sentry Server - Contains authorization metadata. Relatively small.
• Cloudera Navigator Audit Server - Contains auditing information. In large clusters, this database can grow large.
• Cloudera Navigator Metadata Server - Contains authorization, policies, and audit report metadata. Relatively small.

The Cloudera Manager Service Host Monitor and Service Monitor roles have an  internal datastore .  （注意，就是此处说明了， Host Monitor and Service Monitor在CM5版本中，不能配置外部表，只能使用内置表。与CM4版本有区别）

Cloudera Manager 提供三种不同的安装方式，方法A是自动化安装，方法B和C是使用rpm或tar手动安装:

• Path A automatically installs an embedded PostgreSQL database to meet the requirements of the services. This path reduces the number of installation tasks to complete and choices to make. In Path A you can optionally choose to create external databases forActivity Monitor, Reports Manager, Hive Metastore, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server.
• Path B and Path C require you to create databases for the Cloudera Manager Server, Activity Monitor, Reports Manager, Hive Metastore, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server.

使用外部数据库需要更多的输入以及相关工作，但是cloudera提供了更多的兼容性和扩展性，让你可以弹性的选择数据库和配置。

当然可以在一套系统中安装多种不同的数据库，但是这样会带来很多不确定的因素，所以cloudera建议始终使用同一种数据库。

在很多例子中，你需要将相应的service与database安装到同一台机器上，可以减小网络IO，提高整体效率。

当然，你也可以将service和database分开安装到不同的机器上，在大型部署中或者database管理员需要这样的配置，比如这样的场景，Oracle DBA需要独立的管理database。

搭建数据库的配置参考官网，有详细配置步骤：

搭建Cloudera Manager Server数据库

为Activity Monitor, Reports Manager, Hive Metastore, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server搭建外部数据库

为Hue，Oozie搭建外部数据库

下一篇文章中，我将详细介绍Cloudera Manager中database的存储机制，怎样配置，调优等。