[教程] CentOS 7 CDH 5.16 安装教程
一、CDH5.16下载地址
1、cm 地址
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
2、cdh包的地址
http://archive.cloudera.com/cdh5/parcels/5.16.1.3/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.16.1.3/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.16.1.3/manifest.json
3、其他环境
java1.8以上,mysql驱动5.4
二、配置环境
1、机器环境
ip | host | 节点 |
---|---|---|
192.168.239.1 | hadoop01 | 主节点 |
192.168.239.2 | hadoop02 | 子节点1 |
192.168.239.3 | hadoop03 | 子节点2 |
所有机器分别修改本机的hostname
echo "hadoop01" > /etc/hostname
echo "hadoop02" > /etc/hostname
echo "hadoop03" > /etc/hostname
2、修改机器HOST(所有节点)
vi /etc/hosts
192.168.239.1 hadoop01
192.168.239.2 hadoop02
192.168.239.3 hadoop03
3、关闭防火墙(所有节点)
所有机器分别执行
systemctl stop firewalld
systemctl disable firewalld
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
4、配置JDK(所有节点)
删除openjdk
rpm -qa|grep java
rpm -e --nodeps java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64
配置JAVA
复制jdk并且修改权限
mv jdk1.8.0_211 /usr/lib/
chmod 777 -R jdk1.8.0_211/
修改环境变量
vi /etc/profile
加入以下
export JAVA_HOME=/usr/lib/jdk1.8.0_211
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME CLASSPATH PATH
source /etc/profile
5、无密码登录(所有节点)
生成公钥
ssh-keygen -t rsa
每台机器执行复制公钥实现无密码登录
ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03
6、NPT配置
所有节点安装ntp服务
yum -y install ntp
主节点同步一个时间服务器 ntp.api.gz
vi /etc/ntp.conf
server ntp.api.gz iburst
子节点同步为主节点时间服务器
vi /etc/ntp.conf
server hadoop01 iburst
重启npt服务
systemctl restart ntpd
systemctl enable ntpd
7、下载配置缺少的包
yum -y install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security x86_64 spax time zlib-devel
修改部分配置
chmod +x /etc/rc.d/rc.local
echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local
echo 0 > /proc/sys/vm/swappiness
echo never > /sys/kernel/mm/transparent_hugepage/defrag
rcp 修改
yum -y install rpcbind
systemctl start rpcbind
echo "systemctl start rpcbind" >> /etc/rc.d/rc.local
三、配置管理数据库 mysql
1、安装数据库
某个几点安装mysql,这里我使用节点三安装,使用mariadb
yum -y install mariadb-server
修改配置加入默认字符集utf8
vi /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0
character-set-server=utf8 \\添加的行
[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
!includedir /etc/my.cnf.d
重启
systemctl restart mariadb
systemctl enable mariadb
2、初始化数据库
根据提示操作Y或者N
mysql_secure_installation
3、创建数据库
mysql -uroot -p
#执行sql创建cdh所有需要的所有数据库
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
4、修改数据库权限
grant all on *.* to root@"%" Identified by "rootmc";
如果mysql不在本地要把scm的权限也加进去,
grant all on *.* to scm@"%" Identified by "scm";
exit
四、传包解压(所有节点)
把cm安装包复制到/opt目录下
cd /opt
tar vfx cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
复制mysql Jar包到环境里
cp mysql-connector-java-5.1.30-bin.jar /usr/share/java/mysql-connector-java.jar
cp /usr/share/java/mysql-connector-java.jar /opt/cm-5.16.1/share/cmf/lib/
把cdh的parcel放到相应文件夹下
mv manifest.json CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/
把sha1 改名 sha
mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
五、初始化cm
1、初始化CM库
在主节点 把cdh管理的表自动加载到数据库里面
/opt/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop03 -uroot -prootmc --scm-host hadoop03 scm scm scm
-hhadoop03 -uroot -prootmc 为mysql的连接信息
--scm-host hadoop03 scm scm scm 为要创建的数据库scm,用户名scm,密码scm
显示 All done, your SCM database is configured correctly! 表示成功
,如果初始化报错。查看日志,有时候会由于scm的用户没有权限登录mysql
2、创建系统用户(所有节点)
useradd --system --home=/opt/cm-5.16.1/run/cloudera-scm-server --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
修改cm配置文件
sed -i "s/server_host=localhost/server_host=hadoop01/" /opt/cm-5.16.1/etc/cloudera-scm-agent/config.ini
创建允许所有需要文件夹
mkdir /opt/cm-5.16.1/run/cloudera-scm-agent
六、启动
主服务器启动server 子服务器启动agent
/opt/cm-5.16.1/etc/init.d/cloudera-scm-server start
/opt/cm-5.16.1/etc/init.d/cloudera-scm-agent start
访问
192.168.239.1:7180
账户admin 密码admin
七、 补充
设置开机自启
1.复制启动脚本
主节点
cp cloudera-scm-server /etc/init.d/
cp cloudera-scm-agent /etc/init.d/
其他节点
cp cloudera-scm-agent /etc/init.d/
2.修改文件
主节点修改server配置文件路径
vi /etc/init.d/cloudera-scm-server
export JAVA_HOME=xxxxxxx
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cm-5.16.1/etc/default}
所有节点修改agent配置文件路径
vi /etc/init.d/cloudera-scm-agent
CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}
修改成
CMF_DEFAULTS=${CMF_DEFAULTS:-/opt/cm-5.16.1/etc/default}