项目实战：MYSQL数据库集群高可用和数据监控平台

MYSQL集群高可用使用双主双活+keepalived实现
MYSQL数据监控平台使用mysqld_exporter+prometheus+Grafana三件套实现
集群及监控平台搭建完毕后可以实现企业内部的mysql数据库双主机在线增加高可用性，通过keepalived的故障检测和VIP漂移能力使得发生故障后使用者无感知，增加系统的容错能力，通过监控平台实现mysql数据库

1.3.项目特点

综合项目从主从复制原理、数据同步方式、主从架构模式、主从架构/双主架构、高可用、数据可视化一步步从零搭建出一个完善的数据库集群/监控平台
综合项目完成后可以实现对MYSQL数据库高级运维知识的巩固，通过参与实际项目增加工作实战经验，提升技术能力、故障判断检测维护能力、充实简历项目经历
通过prometheus数据库监控三件套的项目搭建，可以掌握数据监控的知识应用能力，通过监控大屏数据的展示使得数据可视化得以实现，使得运维更自动化、直观化

二、项目背景知识

2.1.MySQL中的主从复制技术

`MySQL`是基于它自身的`Bin-log`日志来完成数据的异步复制，因为Bin-log日志中会记录所有对数据库产生变更的语句，包括DML数据变更和DDL结构变更语句，数据的同步过程如下：

2.2.集群机构模式

2.2.1.一主一从/多从架构

一主一从或一主多从，这是传统的主从复制模型，也就是多个主从节点组成的集群中，只有一个主节点，剩余的所有节点都为其附属关系，结构如下：

2.2.2.双主/多主架构

若公司项目中读写请求的比例对半开，同时整体的并发量也不算低，至少超出了单库的承载阈值，这时就可以选用双主/多主架构，结构如下：

2.3.keepalived高可用方案

Keepalived是一个轻量级别的高可用解决方案，使用VRRP（Vritrual Router Redundancy Protocol,虚拟路由冗余协议)的VIP虚拟IP的漂移功能，实现单点故障转移

2.4.可视化监控平台三件套

Mysqld_exporter：是一款轻量级的mysql监控工具，用来收集MysQL数据库相关指标并将其暴露给prometheus进行监控和告警
Prometheus：普罗米修斯，一个开源的服务监控系统，它负责采集和存储应用的监控指标数据，并以可视化的方式进行展示，以便于用户实时掌握系统的运行情况，并对异常进行检测
Grafana：格拉法娜，是一个跨平台的开源的度量分析和可视化工具,可以从prometheus获取数据进行可视化数据大屏展示。
总结：mysqld_exporter用于抓取mysql监控指标数据，prometheus接收到数据后进行整理分析，grafana从prometheus获取数据使用大屏模版进行仪表盘展示

三、项目环境

3.1.软硬件环境清单

主机名	IP地址	硬件	软件
master1	192.168.13.133 VIP:192.168.13.200	cpu:1颗2核内存：2GB HDD：20GB 网络：NAT	VmWare17 OpenEuler22.03 SP4 MySql8.0.37 Keepalived2.2.4
master2	192.168.13.134 VIP:192.168.13.200	cpu:1颗2核内存：2GB HDD：20GB 网络：NAT	VmWare17 OpenEuler22.03 SP4 MySql8.0.37 Keepalived2.2.4
monitor	192.168.13.135	cpu:1颗2核内存：2GB HDD：20GB 网络：NAT	VmWare17 OpenEuler22.03 SP4 Mysqld_Exporter-0.15.1 Prometheus-2.53.2 grafana-enterprise-11.1.2

主机名

IP地址

硬件

软件

master1

192.168.13.133 VIP:192.168.13.200

cpu:1颗2核

内存：2GB

HDD：20GB

网络：NAT

VmWare17

OpenEuler22.03 SP4

MySql8.0.37

Keepalived2.2.4

master2

192.168.13.134 VIP:192.168.13.200

cpu:1颗2核

内存：2GB

HDD：20GB

网络：NAT

VmWare17

OpenEuler22.03 SP4

MySql8.0.37

Keepalived2.2.4

monitor

192.168.13.135

cpu:1颗2核

内存：2GB HDD：20GB

网络：NAT

VmWare17

OpenEuler22.03 SP4

Mysqld_Exporter-0.15.1 Prometheus-2.53.2 grafana-enterprise-11.1.2

四、项目实现步骤

4.1.系统平台部署

安装VmWare17
虚拟出三台计算机
安装OpenEuler22.03 SP4 LTS 操作系统
系统设置：主机名、防火墙、SELinux、hosts映射、IP地址：

4.1.1.系统配置

4.1.2.环境配置：

修改主机名：
[root@localhost ~]# hostnamectl  set-hostname  master1
[root@localhost ~]# hostnamectl  set-hostname  master2
[root@localhost ~]# hostnamectl  set-hostname  monitor

关闭三台主机防火墙及SELinux:
vi  /etc/selinux/config 
SELINUX=disabled

systemctl stop  firewalld        # 关闭防火墙
systemctl disable  firewalld     # 取消开机启动
reboot                           # 重启

三台主机下载所需软件并升级:
yum  install  vim  make  gcc  tree  net-tools tar  -y
yum  update

三台主机重新设置登录系统密码:
passwd root

三台主机时间同步:
vim  /etc/chrony.conf 
# 定位第3行，删除后添加阿里的时间同步服务地址
server   ntp.aliyun.com   iburst

systemctl restart  chronyd

chronyc  sources  -v                 # 多执行几次

timedatectl

4.1.3. 制作三台主机快照

4.2.生产环境二进制包安装MySql

#将mysql二进制包上传至虚拟机并解压缩:

[root@master1 ~]# tar  xvf  mysql-8.0.37-linux-glibc2.17-x86_64.tar.xz

[root@master1 ~]# cd  mysql-8.0.37-linux-glibc2.17-x86_64

[root@master1 mysql-8.0.37-linux-glibc2.28-x86_64]# ls
bin   include  LICENSE  README  support-files  docs  lib  man  share

# 移动到默认安装目录，也可自行修改
[root@master1 ~]# cd  ~ 

[root@master1 ~]# mv  mysql-8.0.37-linux-glibc2.17-x86_64  /usr/local/mysql

[root@master1 /]# cd /usr/local/mysql   

[root@master1 mysql]# groupadd mysql    # 创建名为mysql的用户组

[root@master1 mysql]# useradd -r -g mysql -s /bin/false mysql  # 创建名为 mysql的系统用户，将其添加到mysql用户组中，并设置其登录shell为/bin/false，以限制该用户的登录权限

[root@master1 mysql]# mkdir data   # 创建用于存放MySQL数据文件目录

# 设置mysql目录的账户及工作组，生产环境中不要使用root	
[root@master1 mysql]# chown -R mysql:mysql /usr/local/mysql

4.2.1.初始化软件

[root@master1 mysql]# bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data # 注意：需要复制密码

[root@master1 mysql]# bin/mysqld_safe --user=mysql &  # 使用后台方式以mysql用户身份启动 MySQL 服务器，mysqld_safe 是一个用于启动和监控 MySQL 服务器的脚本

# 注意：此时上述命令执行完毕处于后台运行状态，需要另行启动一个终端

[root@master1 ~]# ps  -ef | grep  mysql   # 查看进程运行状态

[root@master1 ~]# cd  /usr/local/mysql

[root@master1 ~]# bin/mysql -uroot -p   # 登录，可能报错

# 报错，需要找到下面的文件进行软连接
[root@master1 ~]# ln -s /usr/lib64/libncurses.so.6.3 /usr/lib64/libncurses.so.5

[root@master1 ~]# ln -s /usr/lib64/libtinfo.so.6.3 /usr/lib64/libtinfo.so.5

[root@master1 ~]# bin/mysql -u root -p
Enter password:   # 粘贴之前的初始密码

mysql> alter  user 'root'@'localhost'  identified  with  mysql_native_password by '123456';  修改密码

mysql> flush privileges;   # 刷新

mysql> use mysql;  # 查看账户信息

mysql> select user, host, plugin from mysql.user;
+------------------+-----------+-----------------------+
| user             | host      | plugin                |
+------------------+-----------+-----------------------+
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | mysql_native_password |
+------------------+-----------+-----------------------+
4 rows in set (0.00 sec)

mysql>exit

[root@master1 ~]# ps -ef | grep mysql

[root@master1 ~]# kill  -9  pid号   # 在当前终端关闭运行的mysql

4.2.2.设置mysql的配置文件

# 回到之前的终端，敲一个回车，显示进程以杀死
[root@master1 mysql]# vim /etc/my.cnf  # 新建配置文件，输入以下内容：

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
port = 3306
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
socket = /tmp/mysql.sock
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
default-storage-engine=INNODB
log_error = error.log

4.2.3.配置启动脚本

[root@master1 ~]# cd  /usr/local/mysql/support-files

[root@master1 support-files]# cp -a mysql.server /etc/init.d/mysql

[root@master1 support-files]# vim  /etc/init.d/mysql  # 增加=之后的内容
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data

[root@master1 init.d]# cd  ~ 

[root@master1 ~]# vim  ~/.bash_profile   # 设置环境变量需添加如下语句
export PATH=$PATH:/usr/local/mysql/bin

[root@master1 ~]# source  ~/.bash_profile

[root@master1 ~]# systemctl daemon-reload   # 重载系统配置

[root@master1 ~]# systemctl start mysql

[root@master1 ~]# /usr/lib/systemd/systemd-sysv-install enable mysql # 开机启动
  
[root@master1 ~]# mysql  -uroot -p

mysql> update mysql.user set host="%" where user="root";

mysql> flush privileges;

mysql> exit

4.2.4.注意

mysql安装完毕后需要做快照
若选择安装完毕后进行克隆主机作为mater2，则必须要修改master2的server-uuid，否则无法进行下列主从服务器的搭建:

[root@master1 ~]# cat  /usr/local/mysql/data/auto.cnf
[auto]
server-uuid=f15a54e5-5954-11ef-a5b0-000c29daf3cc

# 只需删除master2的/usr/local/mysql/data/auto.cnf文件重启即可重新生成

4.3.msyql集群搭建

搭建主-主集群，两个节点都是主库，也都属于对方的从库，也就是两者之间会相互同步数据，这时为了防止主键出现冲突，一般都会通过设置数据库自增步长的方式来防重
主-主架构集群适用于中读写请求的比例对半开，同时整体的并发量也不算低，至少超出了单库的承载阈值的场景下，架构图如下：

4.3.1.master1节点配置

修改配置文件:

[root@master1 ~]# systemctl stop mysql

[root@master1 ~]# vim /etc/my.cnf 
[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
port = 3306
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
socket = /tmp/mysql.sock
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
default-storage-engine=INNODB
log_error = error.log

# ------主节点配置-------
# 当前节点在集群中的唯一标识
server-id=1

# 开启bin-log日志，并为bin-log日志取个前缀名（有默认值可不写）
log-bin=mysql-bin-log

# 同步复制时过滤的库（主要将一些不需要备份/同步库写进来）
# 也可以通过binlog-do-db=xx1,xx2... 来指定要复制的目标库
binlog-ignore-db=mysql

# 指定bin-log日志的格式为混合模式（默认为statement）
binlog_format=mixed

# 设置单个binlog日志文件的最大容量
max_binlog_size=1024M

# ------从节点配置-------
# 开启relay-log日志（同样可以指定前缀名）
relay_log=mysql-relay-log

# 开启存储过程、函数、触发器等内容的同步功能
log_bin_trust_function_creators=true

# 同步执行跳过一些错误码（防止同步写入时出现错误导致复制中断）
slave_skip_errors=1062

# ------自增序列配置-------
# 设置自增初始值为1
auto_increment_offset=1

# 设置自增步长为2，自增序列为{1、3、5、7、9.....}
auto_increment_increment=2

[root@master1 ~]# systemctl start mysql

[root@master1 ~]# mysql -uroot -p

mysql> create user 'mback'@'%' identified with mysql_native_password by '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> grant replication slave on *.* to 'mback'@'%';
Query OK, 0 rows affected (0.00 sec)

4.3.2.master2节点配置

修改配置文件:

[root@master2 ~]# systemctl stop mysql

[root@master2 ~]# vim /etc/my.cnf 
[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
port = 3306
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
socket = /tmp/mysql.sock
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
default-storage-engine=INNODB
log_error = error.log

# ------主节点配置-------
# 当前节点在集群中的唯一标识
server-id=2

# 开启bin-log日志，并为bin-log日志取个前缀名（有默认值可不写）
log-bin=mysql-bin-log

# 同步复制时过滤的库（主要将一些不需要备份/同步库写进来）
# 也可以通过binlog-do-db=xx1,xx2... 来指定要复制的目标库
binlog-ignore-db=mysql

# 指定bin-log日志的格式为混合模式（默认为statement）
binlog_format=mixed

# 设置单个binlog日志文件的最大容量
max_binlog_size=1024M

# ------从节点配置-------
# 开启relay-log日志（同样可以指定前缀名）
relay_log=mysql-relay-log

# 开启存储过程、函数、触发器等内容的同步功能
log_bin_trust_function_creators=true

# 同步执行跳过一些错误码（防止同步写入时出现错误导致复制中断）
slave_skip_errors=1062

# ------自增序列配置-------
# 设置自增初始值为2
auto_increment_offset=2

# 设置自增步长为2，自增序列为{1、3、5、7、9.....}
auto_increment_increment=2

[root@master2 ~]# systemctl start mysql

[root@master2 ~]# mysql -uroot -p

mysql> create user 'mback'@'%' identified with mysql_native_password by '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> grant replication slave on *.* to 'mback'@'%';
Query OK, 0 rows affected (0.00 sec)

4.3.3.建立master1 节点主从关系

由于建立的是主-主架构集群，相互为对方的从节点，则两个节点都通过root账号登录
此时master1为从节点，master2为主节点
先在master2上查看同步的日志名称及同步点号

# 注意：在master2上执行
mysql> show master status;
+----------------------+----------+--------------+------------------+-------------------+
| File                 | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------+----------+--------------+------------------+-------------------+
| mysql-bin-log.000001 |      664 |              | mysql            |                   |
+----------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

# 日志文件：mysql-bin-log.000001
# 日志数据点：664


#master1 节点上建立复制关系：
mysql> change master to master_host='192.168.13.134',master_user='mback',master_password='123456',master_port=3306,master_log_file='mysql-bin-log.000001',master_log_pos=664;
Query OK, 0 rows affected, 9 warnings (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> show  slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: 192.168.13.134
                  Master_User: mback
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin-log.000001
          Read_Master_Log_Pos: 664
               Relay_Log_File: mysql-relay-log.000002
                Relay_Log_Pos: 330
        Relay_Master_Log_File: mysql-bin-log.000001
             Slave_IO_Running: Yes   # 这里必须为yes
            Slave_SQL_Running: Yes   # 这里必须为yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 664
              Relay_Log_Space: 540
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: 18b551b2-59e8-11ef-a5cd-000c29920312
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

4.3.4.建立master2 节点主从关系

此时master2为从节点，master1为主节点
先在master1上查看同步的日志名称及同步点号

# 注意：在master1上执行
mysql> show master status;
+----------------------+----------+--------------+------------------+-------------------+
| File                 | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------+----------+--------------+------------------+-------------------+
| mysql-bin-log.000001 |      664 |              | mysql            |                   |
+----------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

# 日志文件：mysql-bin-log.000001
# 日志数据点：664


master2节点上建立复制关系：
mysql> change master to master_host='192.168.13.133',master_user='mback',master_password='123456',master_port=3306,master_log_file='mysql-bin-log.000001',master_log_pos=664;
Query OK, 0 rows affected, 9 warnings (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> show  slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: 192.168.13.133
                  Master_User: mback
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin-log.000001
          Read_Master_Log_Pos: 664
               Relay_Log_File: mysql-relay-log.000002
                Relay_Log_Pos: 330
        Relay_Master_Log_File: mysql-bin-log.000001
             Slave_IO_Running: Yes  # 这里必须为yes
            Slave_SQL_Running: Yes  # 这里必须为yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 664
              Relay_Log_Space: 540
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 5bd35595-591a-11ef-abb5-000c29daf3cc
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

4.4.局部测试

测试主主架构是否能相互同步数据

#master1上建立数据库
mysql> create database  test1;
Query OK, 1 row affected (0.01 sec)

mysql> show  databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test1              |
+--------------------+
5 rows in set (0.00 sec)

#在master2上查看是否同步
mysql> show  databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test1              |
+--------------------+
5 rows in set (0.01 sec)

#master2中执行创建表
mysql> use test1;

mysql> create table back_test  (`user_id`int(8) not null,`user_name` varchar(255) not null,`user_sex` varchar(255) not null,`passwordd` varchar(255) not null,`register_time` datetime);
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> desc  back_test;
+---------------+--------------+------+-----+---------+-------+
| Field         | Type         | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| user_id       | int          | NO   |     | NULL    |       |
| user_name     | varchar(255) | NO   |     | NULL    |       |
| user_sex      | varchar(255) | NO   |     | NULL    |       |
| password      | varchar(255) | NO   |     | NULL    |       |
| register_time | datetime     | YES  |     | NULL    |       |
+---------------+--------------+------+-----+---------+-------+
5 rows in set (0.00 sec)

mysql> exit

#在master1上查看是否有同步表数据
mysql> use test1;

mysql> desc  back_test;
+---------------+--------------+------+-----+---------+-------+
| Field         | Type         | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| user_id       | int          | NO   |     | NULL    |       |
| user_name     | varchar(255) | NO   |     | NULL    |       |
| user_sex      | varchar(255) | NO   |     | NULL    |       |
| password      | varchar(255) | NO   |     | NULL    |       |
| register_time | datetime     | YES  |     | NULL    |       |
+---------------+--------------+------+-----+---------+-------+
5 rows in set (0.00 sec)

mysql> exit

4.5.Keepalived故障转移的高可用环境

Keepalived即保持存活，是一种轻量化的高可用方案，用来防止单点故障
Keepalived的工作原理是VRRP（Virtual Router Redundancy Protocol）虚拟路由冗余协议。
Keepalived通过VIP虚拟IP的漂移实现高可用，在相同集群内发送组播包，master1和master2主通过VRRP协议发送组播包，告诉从主的状态。
Keepalived会虚拟出一个VIP地址代理2台服务器向外界提供服务
当Keepalived实际代理master1时，若master1出现故障，keepalived通过自身机制，自动将master2作为实际代理主服务器，不需要人工干预去修改mysql配置信息，从而实现了高可用

4.5.1.master1节点配置

#安装安装包
[root@master1 ~]# yum  install  keepalived -y

#master1节点配置keepalived
[root@master1 ~]# vim  /etc/keepalived/keepalived.conf
# 删除所有，拷贝下列配置数据

! Configuration File for keepalived

global_defs {
   router_id mysql-master01   # keepalived服务器的一个标识，每台机子不同
}

vrrp_instance VI_1 {
    state BACKUP      # 指定keepalived的角色, BACKUP模式将根据优先级决定主或从
    interface ens33   # 监测的网卡名
    virtual_router_id 51  # 虚拟路由标识，确保和master2相同
    priority 100      # 用来选举master的数值
    nopreempt
    advert_int 1
    authentication {    # 认证区域
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {   # 指定VIP地址
        192.168.13.200
    }
}

# 虚拟服务器，需要指定虚拟IP地址和服务端口，IP与端口之间用空格隔开
virtual_server 192.168.13.200 3306 {  
    delay_loop 6            # 设置运行情况检查时间，单位是秒
    lb_algo rr              # 设置后端调度算法
    lb_kind DR              # 设置LVS实现负载均衡的机制
    persistence_timeout 50  # 会话保持时间，单位是秒
    protocol TCP            # 指定转发协议类型
    real_server 192.168.13.133 3306 { # 真实服务器IP地址及端口
    notify_down /etc/keepalived/chk_mysql.sh
        weight 1        # 配置服务节点的权值
        TCP_CHECK {
            connect_port 3306     # 健康检查端口
            connect_timeout 3     # 连接超时时间
            retry 3               # 重连次数
            delay_before_retry 3  # 重连间隔时间
        }
    }
}


# 配置mysql健康检查脚本

[root@master1 ~]# vim  /etc/keepalived/chk_mysql.sh

#!/bin/bash
counter=$(netstat -na|grep "LISTEN"|grep "3306"|wc -l)
if [ "${counter}" -eq 0 ]; then
    killall keepalived
fi

# 通过端口记录数判断mysql是否运行，mysql停止后终止keepalived，当然也可以在抢救一下



[root@master1 ~]# chmod  +x  /etc/keepalived/chk_mysql.sh  # 设置脚本执行权限

[root@master1 ~]# systemctl start keepalived

[root@master1 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
     Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2024-08-14 09:02:49 CST; 15min ago
# 点击q键退出

[root@master1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.133/24 brd 192.168.88.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 192.168.13.200/32 scope global ens32   # 注意会产生新的VIP
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

4.5.2.master2节点配置

#安装安装包
[root@master2 ~]# yum  install  keepalived -y

#master2节点配置keepalived
[root@master2 ~]# vim  /etc/keepalived/keepalived.conf
# 删除所有，拷贝下列配置数据

! Configuration File for keepalived

global_defs {
   router_id mysql-master02     # 注意与master1区分开来
}

vrrp_instance VI_1 {
    state BACKUP      
    interface ens33            # 注意网卡接口名
    virtual_router_id 51  
    priority 50                # 注意选举参数
    nopreempt
    advert_int 1
    authentication {    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {   
        192.168.13.200
    }
}


virtual_server 192.168.13.200 3306 {  
    delay_loop 6            
    lb_algo rr              
    lb_kind DR              
    persistence_timeout 50  
    protocol TCP            
    real_server 192.168.13.134 3306 {    # master2地址及端口
    notify_down /etc/keepalived/chk_mysql.sh
        weight 1        
        TCP_CHECK {
            connect_port 3306     
            connect_timeout 3     
            retry 3               
            delay_before_retry 3  
        }
    }
}


[root@master2 ~]# vim  /etc/keepalived/chk_mysql.sh

#!/bin/bash
counter=$(netstat -na|grep "LISTEN"|grep "3306"|wc -l)
if [ "${counter}" -eq 0 ]; then
    killall keepalived
fi


[root@master2 ~]# chmod  +x  /etc/keepalived/chk_mysql.sh  

[root@master2 ~]# systemctl start keepalived

[root@master2 ~]# systemctl status keepalived

4.5.3.局部测试

原理：当master1服务器宕机后，VIP会自动漂移至master2服务器并继续向外提供mysql服务
步骤：
- 在master1中停用mysql
- 查看VIP是否漂移
- 故障修复

#master1中执行
[root@master1 ~]# systemctl status mysql    # 查看mysql状态
● mysql.service - LSB: start and stop MySQL
     Loaded: loaded (/etc/rc.d/init.d/mysql; generated)
     Active: active (running) since Wed 2024-08-14 09:02:44 CST; 26min ago

[root@master1 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
     Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
     Active: active (running) since Wed 2024-08-14 09:02:49 CST; 26min ago

[root@master1 ~]# ip a   
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.133/24 brd 192.168.88.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 192.168.13.200/32 scope global ens32    # 此时VIP正在监听
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@master1 ~]# systemctl stop mysql   # 模拟宕机

[root@master1 ~]# ip a                  # VIP消失
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.133/24 brd 192.168.88.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@master1 ~]# systemctl status keepalived    # 已停用
○ keepalived.service - LVS and VRRP High Availability Monitor
     Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
     Active: inactive (dead)


#master2执行
[root@master2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:92:03:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.134/24 brd 192.168.88.255 scope global noprefixroute ens32
       valid_lft forever preferred_lft forever
    inet 192.168.13.200/32 scope global ens33  # VIP已经漂移，继续服务
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe92:312/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

#master1中恢复msyql服务和keepalived服务，注意必须先恢复msyql服务
[root@master1 ~]# systemctl start mysql

[root@master1 ~]# systemctl start keepalived

[root@master1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:da:f3:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.133/24 brd 192.168.88.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:feda:f3cc/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
# 即使master1恢复后，VIP不会被强占

4.6.安装监控平台

4.6.1.部署Mysqld_exporter

# 192.168.13.135 monitor主机操作
[root@monitor ~]# ls
anaconda-ks.cfg  mysqld_exporter-0.15.1.linux-amd64.tar.gz

[root@monitor ~]# tar xvf mysqld_exporter-0.15.1.linux-amd64.tar.gz

[root@monitor ~]# mv  mysqld_exporter-0.15.1.linux-amd64  /usr/local/mysqld_exporter
[root@monitor ~]# cd  /usr/local/mysqld_exporter

[root@monitor mysqld_exporter]# ls
LICENSE  mysqld_exporter  NOTICE

[root@monitor mysqld_exporter]# vim  .my.cnf  # 注意为隐藏文件
[client]
user = exporter          # 该账户需要再2台mysql节点新建
password = 123456        # 登录密码
host = 192.168.88.200    # 使用VIP访问
port = 3306


# master1操作
[root@master1 ~]# mysql -uroot -p
Enter password: 
mysql> create user 'exporter'@'%' identified with mysql_native_password by '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> grant process, replication client, select on *.* to 'exporter'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)

mysql> exit

# master2操作，由于设置了主-主集群架构，master1创建账户后会同步到master2，只需查看即可
[root@master2 ~]#  mysql -uroot -p
Enter password: 

mysql> select user ,host from mysql.user;
+------------------+-----------+
| user             | host      |
+------------------+-----------+
| exporter         | %         |    # 已经同步
| mback            | %         |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
+------------------+-----------+
6 rows in set (0.00 sec)

mysql> exit

#配置mysqld_exporter的系统服务
# monitor节点操作
[root@monitor ~]# vim /usr/lib/systemd/system/mysqld_exporter.service
[Unit]
Description=https://prometheus.io

[Service]
Restart=on-failure
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf --web.listen-address=:9104

[Install]
WantedBy=multi-user.target

#刷新服务配置并启动服务
# monitor节点操作
[root@monitor ~]# systemctl daemon-reload

[root@monitor ~]# systemctl start mysqld_exporter.service

[root@monitor ~]# systemctl enable mysqld_exporter.service

[root@monitor ~]# systemctl status mysqld_exporter.service

4.6.2.部署Promethues

Prometheus是一个开源系统监控和警报工具包，最初由SoundCloud构建。自2012年成立以来，许多公司和组织都采用了Prometheus，该项目拥有非常活跃的开发人员和用户社区。它现在是一个独立的开源项目，独立于任何公司维护。
Prometheus于2016年加入了云原生计算基金会，成为继Kubernetes之后的第二个托管项目。
架构图：

# monitor操作
[root@monitor ~]# tar xvf prometheus-2.53.2.linux-amd64.tar.gz

[root@monitor ~]# mv prometheus-2.53.2.linux-amd64  /usr/local/prometheus

[root@monitor ~]# cd /usr/local/prometheus
[root@monitor prometheus]# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool
[root@monitor prometheus]# cd  ~

#新建prometheus账户
[root@monitor ~]# useradd --no-create-home --shell /bin/false prometheus

[root@monitor ~]# chown -R prometheus:prometheus /usr/local/prometheus

#配置prometheus系统服务
# monitor操作
[root@monitor ~]# vim  /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/prometheus/prometheus --config.file /usr/local/prometheus/prometheus.yml --storage.tsdb.path  /usr/local/prometheus/data

[Install]
WantedBy=multi-user.target

#刷新服务配置并启动服务
[root@monitor ~]# systemctl  daemon-reload

[root@monitor ~]# systemctl enable  --now  prometheus

[root@monitor ~]# systemctl  status  prometheus

#重启Prometheus服务
[root@monitor ~]# systemctl restart prometheus

Mysqld_exporter对接Prometheus:

输入http://192.168.13.135:9090 测试是否监控MySql:

4.6.3.部署Grafana

Grafana是一个功能强大、灵活性高、易于使用的数据可视化和监控工具，广泛应用于IT运维、应用性能监控、工业物联网等领域
使用Grafana，用户可以轻松地创建各种图表、图形和面板，以直观和动态的方式展示数据趋势、指标和警报。其灵活的插件系统和丰富的图形化选项使用户能够根据自己的需求定制仪表板，并将其集成到现有的监控系统中

#下载并解压缩：
[root@monitor ~]# yum  install grafana-enterprise-11.1.2-1.x86_64.rpm  -y

#修改配置文件：
[root@monitor ~]# vim  /etc/grafana/grafana.ini
################################# Server #################################
[server]
# Protocol (http, https, h2, socket)
protocol = http       # 启用

# This is the minimum TLS version allowed. By default, this value is empty. Accepted values are: TLS1.2, TLS1.3. If nothing is set TLS1.2 would be taken
;min_tls_version = ""

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3000       # 启用

# The public facing domain name used to access grafana from a browser
domain = localhost      # 启用

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = %(protocol)s://%(domain)s:%(http_port)s/    # 启用

#启动
[root@monitor ~]# systemctl enable --now  grafana-server

[root@monitor ~]# systemctl status grafana-server

#关联Promethues并设置仪表盘模版
# 浏览器其中输入：http://192.168.13.135:3000

# 初始账户/密码：admin/admin

# 设置新的密码：123456

# 在设置界面中选择Home > Connections > Data sources > prometheus

# 设置prometheus监听地址：http://192.168.13.135:9090

# save&test

# 选择监控模版来显示mysql的关键指标，模版ID为7362

4.7.压力测试

查看VIP所在服务器，停止当前节点的mysql，查看VIP是否漂移，刷新监控界面，看是否高可用

MySQL自带的压力测试工具——Mysqlslap

# 修改最大连接数
vim /etc/my.cnf
max_connections=1024

systemctl restart mysql

# 完成压测，查看仪表盘
mysqlslap --defaults-file=/etc/my.cnf --concurrency=200,400 --iterations=1 --number-int-cols=50 --number-char-cols=60 --auto-generate-sql --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --engine=innodb --number-of-queries=2000 -uroot -p123456 --verbose