Centos7 使用yum安装CDH5.14

概述

目标是搭建一个CDH 的测试环境
系统配置如下：

• centos7.3 16G内存，200G磁盘(master节点)
• centos7.3 16G内存，200G磁盘(node节点)
• centos7.3 16G内存，200G磁盘(node节点)
• centos7.3 16G内存，200G磁盘(node节点)
  搭建的cdh为最新的CDH5.14，感觉master内存还是太少，有点吃力

主机配置

192.168.0.1    cdh1
192.168.0.2    cdh2
192.168.0.3    cdh3
192.168.0.4    cdh4

搭建步骤

CDH的搭建有多种方法，一般来说，都是先搭建Cloudera Manager,然后利用Cloudera Manager搭建CDH，如果是测试环境，可以直接利用Cloudera Manager自动化安装，这种方式使用内嵌的PostgreSQL作为metadata等数据的存储，不适于生产环境。生产环境中一般会使用Mysql或其他独立搭建的数据库。

准备工作

在所有节点上都要执行的工作，包括host设置，设置yum repo，无密钥登录，ntp设置，安装jdk,关闭防火墙等等。

• 修改 hostname，执行命令 hostname NAME,并修改 /etc/hostname文件。

• 修改/etc/hosts文件

• 192.168.0.1    cdh1
  192.168.0.2    cdh2
  192.168.0.3    cdh3
  192.168.0.4    cdh4

• 其中cdh1为master节点，接着为每台主机配置hostname（cloudera-scm-agent需要）

• $ hostname cdhxxx
  修改 /etc/sysconfig/network 配置文件，以便下次重启的时，使用新的主机名
  HOSTNAME=cdhxxx
  cdhxxx跟host所在地一致，比如192.168.0.1就是cdh1
  

• 关闭防火墙.
  

• $ systemctl stop firewalld
  $ systemctl disable firewalld

• 所有节点设置无密码登陆

• & yum -y install openssh-clients
  $ ssh-keygen -t rsa    #一直回车
  $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh2
  $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh3
  $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh4
  $ ssh root@cdh2    #测试

  注意：以上所有节点都需要配置
  
• 设置 yum repo

• $ curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
  $ curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo

• ntp设置，安装ntp，编辑crontab定时同步
  
  

  $ yum -y install ntp
  $ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  $ ntpdate time1.aliyun.com
  $ crontab -e

  
  

• 输入:  30 02 * * *  ntpdate time1.aliyun.com
• 安装jdk
• yum install oracle-j2sdk1.7 -y
  
• 关闭SElinux
  
• # To disable SELinux until next reboot:
  setenforce Permissive
• 修改/etc/selinux/config为disabled，重启生效。

  # This file controls the state of SELinux on the system.
  # SELINUX= can take one of these three values:
  #       enforcing - SELinux security policy is enforced.
  #       permissive - SELinux prints warnings instead of enforcing.
  #       disabled - SELinux is fully disabled.
  SELINUX=disabled
  # SELINUXTYPE= type of policy in use. Possible values are:
  #       targeted - Only targeted network daemons are protected.
  #       strict - Full SELinux protection.
  SELINUXTYPE=targeted

  
  
• master节点安装与配置Mysql

  $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
  $ sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
  $ yum update -y
  $ sudo yum install -y mysql-server
  $ sudo systemctl start mysqld

• 删除/var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1文件，并配置/etc/my.conf 
  如:my.conf文件
  
  安装 MySQL JDBC Driver (mysql-connector-java-5.1.44-bin.jar网上找)
  $ mkdir -p /usr/share/java/
  $ cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar
  $ sudo systemctl restart mysqld
  
  创建cdh对应的数据库。
  mysql -u root -p
  mysql> create database amon DEFAULT CHARACTER SET utf8;   #Activity Monitor
  mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';
  
  mysql> create database rman DEFAULT CHARACTER SET utf8;   #Reports Manager
  mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';
  
  mysql> create database metastore DEFAULT CHARACTER SET utf8;  #hive
  mysql> grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';
  
  mysql> create database sentry DEFAULT CHARACTER SET utf8;
  mysql> grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';
  
  mysql> create database nav DEFAULT CHARACTER SET utf8;    #Navigator Audit Server
  mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';
  
  mysql> create database navms DEFAULT CHARACTER SET utf8;  #Navigator Metadata Server
  mysql> grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';
  
  mysql>create database oozie DEFAULT CHARACTER SET utf8;   #Oozie Server
  mysql>grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';
  
  mysql>update user set authentication_string=password('root') where user='root';
  mysql>Grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;
  
  mysql> create database hue DEFAULT CHARACTER SET utf8;  #hue
  mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue_password';
  
  mysql>flush privileges;
  mysql>exit

Cloudera Manager 安装

Cloudera Manager可以说是Cloudera系列产品和组件的核心，它负责统一配置管理CDH集群，基本上本篇文章都是围绕CM的安装部署来展开的，然后再通过CM去安装CDH及其其它相关组件。 
在master节点安装Cloudera Manager Server并启动。
$ yum install -y cloudera-manager-daemons cloudera-manager-server

配置文件目录

/etc/cloudera-scm-server

初始化mysql信息 
[root@cyyy-4 cmf]# whereis cmf
cmf: /usr/lib64/cmf /usr/share/cmf
得到目录/usr/share/cmf
正式初始化

/usr/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -proot --scm-host cdh1 scmdbn scmdbu scmdbp

$ systemctl start cloudera-scm-server

日记
/var/log/cloudera-scm-server

在master和node节点安装Cloudera Manager Agent

CM Agent负责启动和杀死各个CM管理的服务进程，解压和安装CM管理的服务，自动配置，通过心跳机制监视集群服务器状态（默认每15秒Agent发送心跳给CM Server）等
修改 /etc/cloudera-scm-agent/config.ini 中的server_host为master的IP。（192.168.0.1）
$ yum install -y cloudera-manager-agent cloudera-manager-daemons
$ systemctl start cloudera-scm-agent

日记
/var/log/cloudera-scm-agent

访问
http://cdh1:7180

admin admin

hue
http://cdh1:8888

admin admin

停止服务
systemctl stop cloudera-scm-agent
systemctl stop cloudera-scm-server

安装参考：
https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_cdh5_install.html
https://zhangchenchen.github.io/2018/01/26/centos7-install-cdh5.14/