ClouderaManager6.3 集群版搭建

原创 于 2020-03-20 14:05:14 发布 · 1.1k 阅读 · 1 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#cloudera #hadoop #spark #kafka #zookeeper

本文详细介绍了在没有互联网连接的环境下，如何通过离线部署的方式安装Cloudera Manager (CM)及其相关组件，包括系统环境配置、软件包下载、时间同步、主机名设置、SSH免密配置、Java安装、MySQL配置、CM server及agent部署、离线parcel源搭建等步骤。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

1、系统环境

IP主机名核心数内存系统M/S
172.18.216.68cdh-014核16GCentos7.5M
172.18.216.69cdh-024核16GCentos7.5S
172.18.216.70cdh-034核16GCentos7.5S

2、软件包

MySQL JDBC jar

地址：http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar

CM：CM6.3.1

地址：https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz

Parcel：

https://archive.cloudera.com/cdh6/6.3.1/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel

https://archive.cloudera.com/cdh6/6.3.1/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1

https://archive.cloudera.com/cdh6/6.3.1/parcels/manifest.json

 

下载后上传到服务器主节点/opt/cdh-src中：

[root@cdh-01 cdh-src]# ll
total 3414436
-rw-r--r-- 1 root root 2083878000 Mar  5 22:20 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
-rw-r--r-- 1 root root         40 Mar  5 22:05 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
-rw-r--r-- 1 root root 1411444147 Mar  5 22:17 cm6.3.1-redhat7.tar.gz
-rw-r--r-- 1 root root      33887 Mar  5 23:05 manifest.json
-rw-r--r-- 1 root root    1007502 Mar  5 22:17 mysql-connector-java-5.1.47.jar

 

3、ntpd自动同步时间配置（所有节点操作）

# 开启自动同步时间服务，并加入开机启动。
systemctl status ntpd && systemctl start ntpd && systemctl enable ntpd && systemctl status ntpd

4、主机名配置（所有节点操作）

# 注意这里每台机子主机名不一致
hostnamectl set-hostname cdh-0*

5、添加host映射配置（所有节点操作）

#在hosts文件中加入
vim /etc/hosts 
172.18.216.68 cdh-01
172.18.216.69 cdh-02
172.18.216.70 cdh-03

6、SSH免密配置 （所有节点中操作）

#1、检查SSH是否安装，如未安装则：yum install ssh 安装
[root@cdh-01 ~]# rpm -qa |grep ssh
libssh2-1.4.3-10.el7_2.1.x86_64
openssh-clients-7.4p1-16.el7.x86_64
openssh-7.4p1-16.el7.x86_64
openssh-server-7.4p1-16.el7.x86_64

#2、生成SSH秘钥，生成目录在/root/.ssh/ 
[root@cdh-01 ~]#  ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:7TH+Bjatdr/lVDLlfhvWBl3kxc6RXQDbtNbLMBXE81s root@cdh-01
The key's randomart image is:
+---[RSA 2048]----+
|            ..=*O|
|             +.X+|
|            .o+oO|
|         .   .++E|
|        S +.  +o*|
|         o+o.  *o|
|         .o+   oB|
|          o.o .=+|
|         . o..oo.|
+----[SHA256]-----+

#3、在cdh-01中把三台机的/root/.ssh/id_rsa.pub添加到/root/.ssh/authorized_keys文件中去
[root@cdh-01 .ssh]# vim authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAA****QABAAABAQDOpX6vihWM+IcvsARNjuAhZCJql8FE64DKCieKqNkT+Iih/c4RUQuI2YMmaStCjPx82EabneykgF****CosqkNLxFooIo5ApG2YSu0BjNjcicBobk1lSuhxE6yXN3Lo9lhufKRKgNTwUXPc8pVzsunv7cRyo/gHQhlcGH****Oe0H7qtrAffeMaGEBHPXEzdXzyElP0LNXkcVIe3yAEeWYyJ3DBgYSrU/jHopAXR3jX81afOziXerlSPs2tYiR****ELdFGIf6P21NIHcsGxtlM/iLPXSkt5NpJs7rrB4AywemYPqOBmZtW3NLnYXoHoB80s8hR2i+tSvFpErMyc4L root@cdh-01
ssh-rsa AAAAB3NzaC1yc2EAA****QABAAABAQCZrR8J2izPXkd0ZilWjDIPsUUX7uQZt4vCiajBufkgTjRdv3t3ALaU6VC1tmS1Iiqt/cK9r90hNQ****5cAtGMWqeb2Hs+GuA41qEvVeoy2ue1GtBjFrbrKibfj5/AuZE8AA626qyfZl2S+w4QGVoUcErDqkVOmOJZiF1****BJLn/HpOc5v0Ae6+oe0NVetwU9cCqfoy2UwUH+kE2wvZSGlZ6YWJa2xns+xnIlN+UzFfrh/keBObT7uIxrIMV****PkSwHg75eeS0emml6VXLooZvOCKS7rEN/ZrBnW77Z9/+edCRlOkGWfb/TUZdI5yBVh0Q2Vpip+OpRzGXBaLv root@cdh-02
ssh-rsa AAAAB3NzaC1yc2EAA****QABAAABAQC8zUuIppqh6oHAwvG3nJUNzYzl9eq8Iq+tu5vOUT1CNrwVQQkFqJrTJCQ5stA8qYTmmTzS2OhsiM****Pbgkh4dbScpE3EQJYyeXXXCWsGlgeDKTrXNbzZhALz4WkiX5vsmVIG/iq4cvilDX08nrsHaovlcSOJwVzjLDz****qO2ihgJ/hCYdRrh+f2DiL2Q0JBbjFN3eiZyJXrzQFNQA10reTKmXFTJTZp6CJi9ccRVpfZhHpdacYKSaFOvdF****ovK360WQVR8Eknorclxsza0yVwgVXMFG7JC8d/7tKVP6dq71gcDnHArsafEfLOlueMGpuL+9TdKBHvu2S+zf root@cdh-03

#4、拷贝文件/root/.ssh/authorized_keys文件，到另外节点中。
[root@cdh-01 .ssh]# scp ./authorized_keys root@cdh-02:/root/.ssh/
[root@cdh-01 .ssh]# scp ./authorized_keys root@cdh-03:/root/.ssh/


#5、测试连接SSH
[root@cdh-03 ~]# ssh cdh-01
The authenticity of host 'cdh-01 (172.18.216.68)' can't be established.
ECDSA key fingerprint is SHA256:DHlv9hEKuYx8v78pi23kon/9wu+VJE7vXuyx6FiqkBU.
ECDSA key fingerprint is MD5:86:26:05:27:a0:1b:29:4d:5a:51:3f:84:c0:92:84:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cdh-01,172.18.216.68' (ECDSA) to the list of known hosts.
Last login: Mon Nov 11 18:08:55 2019 from 172.18.216.69

Welcome to Alibaba Cloud Elastic Compute Service !

7、安装java （所有节点中操作）

yum install java-1.8.0-openjdk* -y

8、安装mysql （主节点）

安装文档：https://www.cnblogs.com/luohanguo/p/9045391.html

9、创建CDH源数据库、用户（mysql中创建）

CREATE DATABASE cmf DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON cmf.* TO 'cmf'@'%' IDENTIFIED BY 'password';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'password';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'password';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'password';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'password';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'password';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'password';
flush privileges;

10、配置mysql jdbc驱动（主节点）

cp /opt/cdh-src/mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

11、离线部署CM server及agent

  1. 创建目录 （所有节点） 
    mkdir /opt/cdh-src
mkdir /opt/cloudera-manager
  2. 拷贝文件（主节点） 
    mkdir /opt/cdh-src
scp ./cm6.3.1-redhat7.tar.gz root@cdh-02:/opt/cdh-src/
scp ./cm6.3.1-redhat7.tar.gz root@cdh-03:/opt/cdh-src/
  3. 解压文件（所有节点） 
     tar -zxvf cm6.3.1-redhat7.tar.gz -C /opt/cloudera-manager
  4. 选择cdh-01为主节点作为cm server，直接部署（主节点） 
    cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64/
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

     

    1. 选择所有节点作为cm agent，直接部署（所有节点） 
      cd /opt/cloudera-manager/cm6.3.1/RPMS/x86_64
rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm --nodeps --force

       

  5. 修改agent配置，指向server节点（所有节点） 
    sed -i "s/server_host=localhost/server_host=cdh-01/g" /etc/cloudera-scm-agent/config.ini

     

  6. 修改主节点的server配置（主节点） 
    vim /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=cdh-01
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=mypass7789
com.cloudera.cmf.db.setupType=EXTERNAL

     

 

12、主节点部署离线parcel源（主节点）

  1. 安装httpd 
    yum install -y httpd 
systemctl start httpd
  2. 部署离线parcel源  
    mkdir /var/www/html/cdh6_parcel
cp /opt/cdh-src/CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel* /var/www/html/cdh6_parcel/
cp /opt/cdh-src/manifest.json /var/www/html/cdh6_parcel/

[root@cdh-01 cdh6_parcel]# ll
total 2035048
-rw-r--r-- 1 root root 2083878000 Mar  5 23:03 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel
-rw-r--r-- 1 root root         40 Mar  5 23:03 CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1
[root@cdh-01 cdh6_parcel]# mv /opt/cdh-src/manifest.json /var/www/html/cdh6_parcel/
  3. 访问地址查看：http://IP:80/cdh6_parcel

13、启动服务

  1. 启动server服务（主节点） 
    #启动
systemctl start cloudera-scm-server
#启动后会生成日志，日志文件路径/var/log/cloudera-scm-server/，查看启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

    启动成功：

  2. 启动agent服务（所有节点） 
    #启动
systemctl start cloudera-scm-agent
#启动后会生成日志，日志文件路径/var/log/cloudera-scm-agent/，查看启动日志
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

14、打开网页(http://IP:8170)

账号密码：admin/admin

15、选择免费版本

16、创建集群

17、选择免费版本

 

18、等待安装，安装完成后继续

19、这里是CM提示检查主机，这里可以不检查，跳过

 

20、选择要安装的服务

21、进入CM主页面

 

 

 

 

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值