准备安装包:
- mysql环境:
mysql 的rpm包:https://dev.mysql.com/downloads/mysql/
Red Hat Enterprise Linux 7 / Oracle Linux 7 (x86, 64-bit)
jdbc链接包:https://dev.mysql.com/downloads/connector/j/
mysql-connector-java-5.1.45.tar.gz
- CDH包
cloudera manager包:http://archive.cloudera.com/cm5/cm/5/
cloudera-manager-centos7-cm5.13.0_x86_64.tar.gz
CDH包:http://archive.cloudera.com/cdh5/parcels/5.13.0/
CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel
CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1
manifest.json
kafka安装包:http://archive.cloudera.com/kafka/parcels/latest/
csd包:http://archive.cloudera.com/csds/kafka/
1、修改主机名:
hostnamectl --static set-hostname node1
vi /etc/hosts
添加:
10.1.1.240 node1
10.1.1.241 node2
10.1.1.242 node3
10.1.1.243 node4
2、时间同步:
网络配置问题:如果不能使用yum等外网命令,添加DNS1=8.8.8.8
yum -y install ntp
ntpdate -s pool.ntp.org
service ntpd start
3、关闭防火墙、selinux
systemctl status firewalld
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config ##重启生效
- 设置系统swappiness参数在内存与交换分区:
cat /proc/sys/vm/swappiness
临时设置:sysctl vm.swappiness=8
永久设置:vi /etc/sysctl.conf
添加:vm.swappiness=8
激活 :sysctl -p
echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >>/etc/rc.d/rc.local
4、配置ssh无密码登陆
vi /etc/ssh/sshd_config
去掉下面两行的注释
RSAAuthentication yes PubkeyAuthentication yes
命令:
- ssh-keygen -t rsa ,一直回车
生成id_rsa id_rsa.pub文件
- 在/root/.ssh/下,id_rsa.pub文件内容复制到新文件authorized_keys文件内,
各节点的id_rsa.pub信息都复制到同一个authorized_keys中,各个服务器都复制一份文件authorized_keys文件,内容如下:
- 在操作节点 ssh node1-4节点
生成known_hosts 文件,将该文件复制到各个节点,内容如下:
scp known_hosts node2:/root/.ssh/
5、卸载系统自带的JDK
rpm -qa | grep jdk
yum -y remove java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64
卸载前三个
复制JDK :
scp jdk-8u151-linux-x64.rpm node4:/home/package/
安装JDK:
rpm -ivh jdk-8u151-linux-x64.rpm
6、安装MYSQL
tar -xvf mysql-5.7.20-1.el7.x86_64.rpm-bundle.tar
- 卸载有冲突的组件:
yum -y remove mariadb-libs-5.5.52-1.el7.x86_64
- 需要安装的包:
rpm -ihv mysql-community-client-5.7.20-1.el7.x86_64.rpm
mysql-community-common-5.7.20-1.el7.x86_64.rpm
mysql-community-libs-5.7.20-1.el7.x86_64.rpm
mysql-community-server-5.7.20-1.el7.x86_64.rpm
datadir =datadir=/home/mysqldata
mysqld --initialize --user=mysql
- #启动mysql
service mysqld start
设置开机启动:
systemctl enable mysqld
systemctl list-unit-files | grep mysqld
- 查看root密码:
grep 'temporary password' /var/log/mysqld.log
重置密码:
SET PASSWORD=PASSWORD('mysqldb'); --localhost 的root,权限最大
允许远程访问:
grant all on *.* to root@"%" Identified by "mysqldb"; ---全IP的root,需要接受上面root的授权
flush privileges;
mysql -uroot -pmysqldb -----本地登录权限最大
- 创建CDH所需库:
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --hive
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --hue
create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci; --
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
- 创建并初始化cloudera manager的库
create database scm DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
CREATE USER 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'mysqldb' WITH GRANT OPTION;
flush privileges;
grant all on scm.* to 'scm'@'%'
flush privileges;
/opt/cm-5.13.0/share/cmf/schema/scm_prepare_database.sh mysql -hnode1 -pmysqldb --scm-hostnode1 scm scm scm
说明:
--scm-host node1:CMS的主机,一般是和mysql安装的主机是在同一个主机上。
scm scm scm : 是数据库、用户名、密码
- 安装依赖包:
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs RedHat-lsb cyrus-sasl-plain cyrus-sasl-gssapi
- 创建用户:
useradd -r -d /opt/cloudera-manager/cm-5.13.0/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm
tar -axvf cloudera-manager-centos7-cm5.8.3_x86_64.tar.gz -C /opt/cloudera-manager
vi /opt/cm-5.13.0/etc/cloudera-scm-agent/config.ini
的server_host 为node1
复制并重命名JDBC
cp mysql-connector-java-5.1.45-bin.jar /opt/cm-5.13.0/share/cmf/lib/mf/lib/mysql-connector-java.jar
- 所有节点:
cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/
scp mysql-connector-java-5.1.45-bin.jar node4:/usr/share/java/
- 主节点
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/
cp CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/
mv CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha1 CDH-5.13.0-1.cdh5.13.0.p0.29-el7.parcel.sha
- 主节点启动:
/opt/cm-5.13.0/etc/init.d/cloudera-scm-server start
启动后停止,则是scm库没创建好,重新创建,删除
/opt/cm-5.13.0/run/cloudera-scm-server下的.pid文件
/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent start---坑,不启动主节点不能安装
将文件夹 cm-5.13.0复制都各个节点:
scp -r cm-5.13.0/ node2:/opt/
各副节点:
/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent start
/opt/cm-5.13.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hnode1 -pmysqldb -P3306 --scm-host node1 scm scm scm JAVA_HOME=/usr/java/jdk1.8.0_151
mkdir dfs zookeeper yarn solr oozie cloudera-host-monitor impala hue logs
mkdir dn nn snn
chown -R cloudera-scm:cloudera-scm /home/
- 主机点问题:
http://30.0.0.240:7180/cmf/login
问题:
主机运行状况不良:
rm -f /opt/cm-5.13.0/lib/cloudera-scm-agent/cm_guid
/opt/cm-5.13.0/etc/init.d/cloudera-scm-agent restart
cloudera manager 安装集群失败后,可删除scm数据库,重建,再重新安装
HDFS 启动失败 :
chown -R hdfs:hdfs /home/ ---数据节点根目录
manager客户端启动失败:
chown -R cloudera-scm:cloudera-scm /home/ --cloudera manager 节点目录
- 关闭桥接网卡
brctl show
virsh net-undefine default
systemctl stop libvirtd.service
reboot
Cloudera Manager Agent 无法确定双工模式或接口速度。
Cloudera Manager 的“主机”→“配置”→“监控”
调整为路由器带宽和网卡接口带宽中,低的那个