大数据基础学习笔记（安装CDH的环境配置）-CDH-6.0.0

本文详细介绍了CDH6.0集群的部署步骤，包括软件下载、网络配置、SSH免密登录设置、防火墙及SELinux关闭、JDK安装、NTP配置、MySQL安装与配置、第三方依赖包安装、repo安装、GPG key导入等关键环节。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.下载软件地址(（以centos7.4）)
CDH安装包地址：
https://archive.cloudera.com/cdh6/6.0.0/parcels/
CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel
CDH-6.0.0-1.cdh6.0.0.p0.537114-el7.parcel.sha256
manifest.json

Cloudera Manager下载地址:
https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/

cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm

cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm

cloudera-manager-server-6.0.0-530873.el7.x86_64.rpm

cloudera-manager-server-db-2-6.0.0-530873.el7.x86_64.rpm

oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

2.网络配置

vim /etc/sysconfig/network

vim /etc/hosts

（内网）

3.SSH免密匙登录

1.生成RSA秘钥对
   在XShell下方输入框中输入命令ssh-keygen -t rsa，然后回车执行。再回车3次，即可生成RSA秘钥对
2.查看密钥对文件
   ls .ssh
   id_rsa 私钥
   id_rsa.pub 公钥
3.将每个节点的公钥拷贝到authorized_keys
   可以通过命令ssh-copy-id，将本机公钥复制到其他节点的authorized_keys目录，这样本机即可免秘钥登录其他节点。
      （按照提示输入“yes”和密码，即可将node1的公钥复制到node1、node2和node3节点。（复制本机，可以实现免密登录本机））
      ssh-copy-id node1
      ssh-copy-id node2
      ssh-copy-id node3
      在三个节点都执行
4.测试免密登录
    ssh node1
   ssh node2
   ssh node3
   在三个节点中查看ls .ssh ，ll .ssh，cat .ssh/authorized_keys，cat .ssh/known_hosts
5.退出ssh登录
     exit

4.防火墙关闭

systemctl stop firewalld.service
systemctl disable firewalld.service

service iptables stop
chkconfig iptables off

（本人是买的阿里云服务器，查看关闭防火墙地址https://www.jianshu.com/p/4a9958d2e859）

5.selinux关闭

setenforce 0 （临时生效）
修改 /etc/selinux/config 下的 SELINUX=disabled （重启后永久生效）

6.安装jdk1.8，环境变量

JAVA_HOME=/usr/java/jdk1.8.0_161
PATH=$JAVA_HOME/bin:$PATH
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

立即生效source /etc/profile

检查是否成功java -version

或者安装oracle

yum install oracle-j2sdk1.8

7.安装NTP

yum install ntp 安装NTP
systemctl enable ntpd.service 自启动
pgrep ntpd //查看相关进程
service ntpd status 查询ntpd的状态
ntpdate -u ntp.sjtu.edu.cn 时间的误差

配置NTP服务（所有节点）
集群中所有主机必须保持时间同步，如果时间相差较大会引起各种问题。
具体思路如下：
master节点作为ntp服务器与外界对时中心同步时间，随后对所有slave节点提供时间同步服务。
所有slave节点以master节点为基础同步时间。
所有节点安装相关组件：yum install ntp。完成后，
配置开机启动：chkconfig ntpd on,检查是否设置成功：chkconfig --list ntpd其中2-5为on状态就代表成功。

主节点配置
在配置之前，先使用ntpdate手动同步一下时间，免得本机与对时中心时间差距太大，使得ntpd不能正常同步。
这里选用0.asia.pool.ntp.org作为对时中心,ntpdate 0.asia.pool.ntp.org。
ntp服务只有一个配置文件（/etc/ntp.conf），配置好了就OK。
这里只给出有用的配置，不需要的配置都用#注掉，这里就不在给出：

driftfile /var/lib/ntp/drift

restrict 127.0.0.1

restrict -6 ::1

restrict default nomodify notrap

server ntp1.aliyun.com prefer

minpoll 6

includefile /etc/ntp/crypto/pw

keys /etc/ntp/keys

配置文件完成，保存退出，启动服务，执行如下命令：service ntpd start
检查是否成功，用ntpstat命令查看同步状态，出现以下状态代表启动成功：

如果出现异常请等待几分钟，一般等待5-10分钟才能同步。

配置ntp客户端（所有slave节点）

driftfile /var/lib/ntp/drift

restrict 127.0.0.1

restrict -6 ::1

restrict default kod nomodify notrap nopeer noquery

restrict -6 default kod nomodify notrap nopeer noquery

#这里是主节点的主机名或者ip

server cdh-master.test.com

minpoll 6

includefile /etc/ntp/crypto/pw

keys /etc/ntp/keys

保存退出，请求服务器前，请先使用ntpdate手动同步一下时间：
ntpdate -u node1 (主节点ntp服务器)
启动服务：
service ntpd start

如果时间同步不成功可以看看网址http://blog.itpub.net/26524307/viewspace-1063924/

8.安装mysql

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm

rpm -ivh mysql-community-release-el7-5.noarch.rpm

yum update

yum install mysql-server

systemctl start mysqld

systemctl enable mysqld

Cloudera Manager必须使用innodb引擎。使用MyISAM服务无法启动。

官方推荐配置文件

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES

If you are using an Oracle database for Cloudera Manager Server, edit the /etc/default/cloudera-scm-server file on the Cloudera Manager server host. Locate the line that begins with export CM_JAVA_OPTS and change the -Xmx2G option to -Xmx4G.

mysql jdbc Driver 安装mysql-connector-java-5.1.46.tar.gz版本的

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

tar zxvf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/

cd mysql-connector-java-5.1.46

cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

在数据库创建数据库如下图

手动创建需要建的库有

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm@123';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY ' amon@123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman@123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue@123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore@123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry@123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav@123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms@123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie@123';

或者使用CDH的语法创建

/opt/cloudera/cm/schema/scm_prepare_database.sh \

[options] <databaseType> <databaseName> <databaseUser> <password>

运行/opt/cloudera/cm/schema/scm_prepare_database.sh 可以查看参数

如果数据库和CM在一台服务器上

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

如果数据库和CM不在一台服务器上

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm

如果你的数据库是oracle

/opt/cloudera/cm/schema/scm_prepare_database.sh -h cm-oracle.example.com oracle orcl sample_user sample_pass

9.下载第三方依赖包
在线安装命令 yum install
需要安装的第三方依赖包
chkconfig
python
bind-utils
psmisc
libxslt
zlib
sqlite
fuse
fuse-libs
redhat-lsb
cyrus-sasl-plain
cyrus-sasl-gssapi

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi

yum install python-pip

pip install --upgrade psycopg2

yum install portmap

10.安装repo(所有节点)

wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/

11.导入GPG key（所有节点）

rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera

12.角色分配官方推荐：

13.cdh6 使用的组件版本详情

Cloudera Enterprise 6.0中的主要组件的升级：
SDX
Cloudera Manager 6.0
Cloudera Director 6.0
Cloudera Navigator 6.0
Cloudera Navigator Key Trustee 6.0
Apache Sentry 2.0
Apache Kafka 1.0
Analytics and Machine Learning Workloads
Apache Solr 7.0
Apache Spark 2.3
Core Platform
Apache Hadoop 3.0
Apache Hive 2.1
Apache HBase 2.0
Apache Oozie 5.0
Apache Avro 1.8
Apache Parquet 1.9