CDH5.13安装

本文详细介绍了如何部署Cloudera Manager及其相关组件,包括MySQL、JDK、HDFS等关键步骤,并提供了常见问题解决方案。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

准备安装包:

  • mysql环境:

mysql 的rpm包:https://dev.mysql.com/downloads/mysql/

Red Hat Enterprise Linux 7 / Oracle Linux 7 (x86, 64-bit)

jdbc链接包:https://dev.mysql.com/downloads/connector/j/

mysql-connector-java-5.1.45.tar.gz

  • CDH包

cloudera manager包:http://archive.cloudera.com/cm5/cm/5/

cloudera-manager-centos7-cm5.13.0_x86_64.tar.gz

CDH包:http://archive.cloudera.com/cdh5/parcels/5.13.0/

CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel

CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1

manifest.json

kafka安装包:http://archive.cloudera.com/kafka/parcels/latest/

 csd包:http://archive.cloudera.com/csds/kafka/

 

 

1、修改主机名:

hostnamectl --static set-hostname node1

vi /etc/hosts

添加:

10.1.1.240 node1

10.1.1.241 node2

10.1.1.242 node3

10.1.1.243 node4

2、时间同步:

网络配置问题:如果不能使用yum等外网命令,添加DNS1=8.8.8.8

yum -y install ntp

ntpdate -s pool.ntp.org

service ntpd start

3、关闭防火墙、selinux

systemctl status firewalld

systemctl stop firewalld

systemctl disable firewalld

sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  ##重启生效

 

  • 设置系统swappiness参数在内存与交换分区:

cat /proc/sys/vm/swappiness

临时设置:sysctl vm.swappiness=8

永久设置:vi /etc/sysctl.conf

添加:vm.swappiness=8

激活 :sysctl -p

echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/defrag

 

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >>/etc/rc.d/rc.local

4、配置ssh无密码登陆

vi /etc/ssh/sshd_config

去掉下面两行的注释

RSAAuthentication yes PubkeyAuthentication yes

 

命令:

  • ssh-keygen -t rsa ,一直回车

生成id_rsa id_rsa.pub文件

  • 在/root/.ssh/下,id_rsa.pub文件内容复制到新文件authorized_keys文件内,

各节点的id_rsa.pub信息都复制到同一个authorized_keys中,各个服务器都复制一份文件authorized_keys文件,内容如下:

 

  • 在操作节点 ssh node1-4节点

生成known_hosts 文件,将该文件复制到各个节点,内容如下:

scp known_hosts node2:/root/.ssh/

 

5、卸载系统自带的JDK

rpm -qa | grep jdk

yum -y remove java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64

卸载前三个

复制JDK :

scp jdk-8u151-linux-x64.rpm node4:/home/package/

安装JDK:

rpm -ivh jdk-8u151-linux-x64.rpm

6、安装MYSQL

tar -xvf mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar

  • 卸载有冲突的组件:

yum -y remove mariadb-libs-5.5.52-1.el7.x86_64

  • 需要安装的包:

rpm -ihv mysql-community-client-5.7.20-1.el7.x86_64.rpm

mysql-community-common-5.7.20-1.el7.x86_64.rpm

mysql-community-libs-5.7.20-1.el7.x86_64.rpm

mysql-community-server-5.7.20-1.el7.x86_64.rpm

datadir =datadir=/home/mysqldata

 

mysqld --initialize --user=mysql

  • #启动mysql

service mysqld start

设置开机启动:

 systemctl enable mysqld

systemctl list-unit-files | grep mysqld

  • 查看root密码:

grep 'temporary password' /var/log/mysqld.log

重置密码:

SET PASSWORD=PASSWORD('mysqldb'); --localhost 的root,权限最大

允许远程访问:

grant all on *.* to root@"%" Identified by "mysqldb"; ---全IP的root,需要接受上面root的授权

flush privileges;

 

mysql -uroot -pmysqldb -----本地登录权限最大

  • 创建CDH所需库:

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --hive

create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --hue

create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --

create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

  • 创建并初始化cloudera manager的库

create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

CREATE USER 'scm'@'%' IDENTIFIED BY 'scm';

GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'mysqldb' WITH GRANT OPTION;

flush privileges;

grant all on scm.* to 'scm'@'%'

flush privileges;

/opt/cm-5.13.0/share/cmf/schema/scm_prepare_database.sh mysql -hnode1 -pmysqldb --scm-hostnode1 scm scm scm

说明:

--scm-host node1:CMS的主机,一般是和mysql安装的主机是在同一个主机上。

scm scm scm : 是数据库、用户名、密码

  • 安装依赖包:

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs RedHat-lsb cyrus-sasl-plain cyrus-sasl-gssapi

 

  • 创建用户:

useradd -r -d /opt/cloudera-manager/cm-5.13.0/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm

tar -axvf cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz -C /opt/cloudera-manager

vi /opt/cm-5.13.0/etc/cloudera-scm-agent/config.ini

的server_host 为node1

 

复制并重命名JDBC

cp mysql-connector-java-5.1.45-bin.jar /opt/cm-5.13.0/share/cmf/lib/mf/lib/mysql-connector-java.jar

  • 所有节点:

cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/

scp mysql-connector-java-5.1.45-bin.jar node4:/usr/share/java/

  • 主节点

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

cp CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/

mv CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1 CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha

 

  • 主节点启动:

/opt/cm-5.13.0/etc/init.d/cloudera-scm-server start

启动后停止,则是scm库没创建好,重新创建,删除

/opt/cm-5.13.0/run/cloudera-scm-server下的.pid文件

 

/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent start---坑,不启动主节点不能安装

将文件夹 cm-5.13.0复制都各个节点:

scp -r cm-5.13.0/ node2:/opt/

各副节点:

/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent start

 

/opt/cm-5.13.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hnode1 -pmysqldb -P3306 --scm-host node1 scm scm scm JAVA_HOME=/usr/java/jdk1.8.0_151

 

mkdir dfs zookeeper yarn solr oozie cloudera-host-monitor impala hue logs

mkdir dn nn snn

chown -R cloudera-scm:cloudera-scm /home/

  • 主机点问题:

 

 

 

http://30.0.0.240:7180/cmf/login

问题:

主机运行状况不良:

rm -f /opt/cm-5.13.0/lib/cloudera-scm-agent/cm_guid

/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent restart

 

cloudera manager 安装集群失败后,可删除scm数据库,重建,再重新安装

 

HDFS 启动失败 :

chown -R hdfs:hdfs /home/ ---数据节点根目录

manager客户端启动失败:

chown -R cloudera-scm:cloudera-scm /home/ --cloudera manager 节点目录

  • 关闭桥接网卡

brctl show

virsh net-undefine default

systemctl stop libvirtd.service

reboot

 

 

Cloudera Manager Agent 无法确定双工模式或接口速度。

 

Cloudera Manager 的“主机”→“配置”→“监控”

调整为路由器带宽和网卡接口带宽中,低的那个

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值