作业五

Mysql高可用集群搭建(三) MHA VipHa 故障转移恢复 MHA VIP漂移配置 常见生产故障解决 阅读详情

一、如果主节点已经运行了一段时间,且有大量数据时,如何配置并启动slave节点(写出操作步骤)

  1. 首先备份master节点数据
  2. 恢复数据到slave节点
  3. 复制起始位置为主节点备份时的二进制日志文件

#在主节点做完全备份

[root@CentOS8 ~]# mysqldump -A -F --single-transaction --master-data=1  > /backup/fullback_`date +%F_%T`.sql

[root@CentOS8 ~]# ll /backup/

total 2988

-rw-r--r-- 1 root root 3056439 Oct 17 09:43 fullback_2020-10-17_09:43:38.sql

#从主节点拷贝备份文件到slave节点

[root@CentOS8 ~]# scp /backup/fullback_2020-10-17_09\:43\:38.sql 10.0.0.28:/data/

#将完全备份还原到slave节点

#在slave节点安装mysql

[root@CentOS8 ~]# dnf -y install mariadb-server

添加修改配置文件

[root@CentOS8 ~]# vim /etc/my.cnf.d/mariadb-server.cnf

[mysqld]

server-id=11

read-only

#启动服务

[root@CentOS8 ~]# systemctl start mariadb

#检查完全备份的二进制日志文件及起始POS

[root@CentOS8 ~]# grep '^CHANGE MASTER' /data/fullback_2020-10-17_09\:43\:38.sql

CHANGE MASTER TO MASTER_LOG_FILE='mariadb-bin.000003', MASTER_LOG_POS=389;

#导入完全备份数据到slave节点

[root@CentOS8 ~]# mysql < /data/fullback_2020-10-17_09\:43\:38.sql

#配置slave节点,从完全备份的位置之后开始同步复制

[root@CentOS8 ~]# mysql

MariaDB [(none)]> change master to

    -> master_host='10.0.0.8',

    -> master_user='repluser',

    -> master_password='123456',

    -> master_port=3306,

    -> MASTER_LOG_FILE='mariadb-bin.000003',

    -> MASTER_LOG_POS=389;

Query OK, 0 rows affected (0.003 sec)

#开启slave节点复制

MariaDB [(none)]> start slave;

Query OK, 0 rows affected (0.001 sec)

 

 

二、当master服务器宕机,提升一个slave成为新的master(写出操作步骤)

 

#首先查询slave服务器的relay-log.info信息,找到哪个服务器的数据库是最新,让它成为新master

[root@CentOS8 ~]# cat /var/lib/mysql/relay-log.info

./mariadb-relay-bin.000005

27378298

mariadb-bin.000003

27378179

#新选择的master修改配置文件,关闭read-only配置,并启用二进制日志

[root@slave1 ~]#vim /etc/my.cnf.d/mariadb-server.cnf

[mysqld]

server-id=18

read-only=OFF

log-bin

#重启服务,生成二进制日志文件

[root@CentOS8 ~]# systemctl restart mariadb

MariaDB [(none)]> show master logs;

+--------------------+-----------+

| Log_name           | File_size |

+--------------------+-----------+

| mariadb-bin.000001 |       379 |

| mariadb-bin.000002 |       375 |

+--------------------+-----------+

 

#在新选择的master服务器上清除旧的master信息

MariaDB [(none)]> stop slave;

MariaDB [(none)]> reset slave all;

#在新选择的master服务器上做完全备份

[root@CentOS8 ~]# mysqldump -A  -F --single-transaction --master-data=1 > backup.sql

#如果旧的master服务器没有完全宕机,还能提取出二进制日志文件,那么分析旧的master 的二进制日志,将未同步到至新master的二进制日志导出来,恢复到新master,尽可能的恢复数据

#将新选择的master服务器上做的完全备份拷贝到其他slave服务器上

[root@CentOS8 ~]# scp backup.sql 10.0.0.28:

#在其它所有slave上重新还原数据库,指向新的master

[root@CentOS8 ~]#vim backup.sql   #编辑完全备份文件,添加master信息

CHANGE MASTER TO 

MASTER_HOST='10.0.0.18',   

MASTER_USER='repluser',

MASTER_PASSWORD='123456',

MASTER_PORT=3306,

MASTER_LOG_FILE=' mariadb-bin.000002 ',

MASTER_LOG_POS=375

 

MariaDB [hellodb]>stop slave;

MariaDB [hellodb]>reset slave all;

MariaDB [hellodb]>set sql_log_bin=off;

MariaDB [hellodb]>source backup.sql;

MariaDB [hellodb]>set sql_log_bin=on;

MariaDB [hellodb]>start slave;

 

 

三、通过 MHA 0.58 搭建一个数据库集群结构

 

环境:4台主机

10.0.0.7 CentOS7 MHA管理端 mysql5.7

10.0.0.8 CentOS8 Master mysql8.0

10.0.0.18 CentOS8 Slave1 mysql8.0

10.0.0.28 CentOS8 Slave2 mysql8.0

 

1.安装MHA 0.58

说明: mha4mysql-manager-0.56-0.el6.noarch.rpm 不支持CentOS 8,只支持CentOS7 以下版本

说明: mha4mysql-manager-0.58-0.el7.centos.noarch.rpm ,支持MySQL 5.7 ,但和CentOS8版本上的Mariadb -10.3.17不兼容

共两个包:

mha4mysql-manager

mha4mysql-nod

1.1 在管理节点安装这两个包

[root@mha-manager ~]#yum -y install mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

[root@mha-manager ~]#yum -y install mha4mysql-node-0.58-0.el7.centos.noarch

 

1.2 在3台mysql服务器上安装mha4mysql-node包,此包支持CentOS 8/7/6

[root@CentOS8 ~]# yum -y install mha4mysql-node-0.58-0.el7.centos.noarch.rpm

2. 在所有节点实现相互之间ssh key验证

[root@mha-manager ~]#ssh-keygen

[root@mha-manager ~]#ssh-copy-id  10.0.0.7

[root@mha-manager ~]#rsync -av .ssh 10.0.0.8:/root/    #保留属性,或用scp -p

[root@mha-manager ~]#rsync -av .ssh 10.0.0.18:/root/

[root@mha-manager ~]#rsync -av .ssh 10.0.0.28:/root/

 

3. 在管理节点建立配置文件和相关脚本

3.1 配置文件

[root@mha-manager ~]#mkdir /etc/mastermha/

[root@mha-manager ~]#vim /etc/mastermha/app1.cnf

[server default]

user=mhauser   #管理 MHA 用户的名称

password=123456  #管理 MHA 用户的密码

manager_workdir=/data/mastermha/app1/  #MHA manager 使用的工作目录

manager_log=/data/mastermha/app1/manager.log  #MHA manager 生成日志的保存路径及文件名

remote_workdir=/data/mastermha/app1/  #每一个 MHA node 保存从 master 节点复制而来的二进制日志的工作目录

ssh_user=root  MHA manager 和 MHA node 通过此用户连接至 Mysql 所在的主机

repl_user=repluser  #主从复制时使用的用户名称

repl_password=123456   主从复制时使用的用户密码

ping_interval=1  MHA manager ping master 节点的时间间隔,单位为秒,当连续 3 次 ping 失败后, MHA manager 认为此 master 节点宕机

master_ip_failover_script=/usr/local/bin/master_ip_failover  #故障转移脚本

report_script=/usr/local/bin/sendmail.sh  #以邮件告警的方式来发送 failover 报告脚本

check_repl_delay=0  #默认情况下如果一个 slave 节点落后 master 节点 100M 的中继日志, MHA 将不会选择该 slave 节点作为一个新的 master 节点,因为对于这个 slave 节点的恢复需要花费很长时间,通过设置 check_repl_delay=0 , MHA 在选择一个新的 master 节点时将会忽略复制延时,此参数对于设置了 candidate_master=1 的主机非常有用,因为此备选 master 节点在切换过程中一定是新的 master 节点

master_binlog_dir=/data/mysql/  #master 节点生成二进制日志的目录

[server1]

hostname=10.0.0.8

candidate_master=1  #从不同的 slave 节点中,提升一个可靠的节点作为新的 master 节点,如果此值设置为 1 ,此 slave 节点会优先成为新的 master 节点,即使这个 slave 节点的事件不是最新的

[server2]

hostname=10.0.0.18

candidate_master=1

[server3]

hostname=10.0.0.28

 

3.2 相关脚本

#报警邮件脚本

[root@mha-manager ~]#cat /usr/local/bin/sendmail.shecho "MySQL is down" | mail -s "MHA Warning" root@xxx.com

[root@mha-manager ~]#chmod +x /usr/local/bin/sendmail.

#故障主备切换脚本

#!/usr/bin/env perl

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

my (

$command, $ssh_user, $orig_master_host, $orig_master_ip,

$orig_master_port, $new_master_host, $new_master_ip, $new_master_port

);

my $vip = '10.0.0.100';#设置Virtual IP

my $gateway = '10.0.0.2';#网关Gateway IP

my $interface = 'eth0';

my $key = "1";

my $ssh_start_vip = "/sbin/ifconfig $interface:$key $vip;/sbin/arping -I $interface -c 3 -s $vip $gateway >/dev/null 2>&1";

my $ssh_stop_vip = "/sbin/ifconfig $interface:$key down";

GetOptions(

'command=s' => \$command,

'ssh_user=s' => \$ssh_user,

'orig_master_host=s' => \$orig_master_host,

'orig_master_ip=s' => \$orig_master_ip,

'orig_master_port=i' => \$orig_master_port,

'new_master_host=s' => \$new_master_host,

'new_master_ip=s' => \$new_master_ip,

'new_master_port=i' => \$new_master_port,

);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

# $orig_master_host, $orig_master_ip, $orig_master_port are passed.

# If you manage master ip address at global catalog database,

# invalidate orig_master_ip here.

my $exit_code = 1;

eval {

print "Disabling the VIP on old master: $orig_master_host \n";

&stop_vip();

$exit_code = 0;

};

if ($@) {

warn "Got Error: $@\n";

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "start" ) {

# all arguments are passed.

# If you manage master ip address at global catalog database,

# activate new_master_ip here.

# You can also grant write access (create user, set read_only=0, etc) here.

my $exit_code = 10;

eval {

print "Enabling the VIP - $vip on the new master - $new_master_host \n";

&start_vip();

$exit_code = 0;

};

if ($@) {

warn $@;

exit $exit_code;

}

exit $exit_code;

}

elsif ( $command eq "status" ) {

print "Checking the Status of the script.. OK \n";

`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;

exit 0;

}

else {

&usage();

exit 1;

}

}

# A simple system call that enable the VIP on the new master

sub start_vip() {

`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;

}

# A simple system call that disable the VIP on the old_master

sub stop_vip() {

`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

}

sub usage {

print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

}

 

4.配置master 服务器

[root@master ~]# yum -y install mysql-server

[root@master ~]#mkdir /data/mysql/

[root@master ~]#chown mysql.mysql /data/mysql

#准备配置文件

[root@master ~]#vim /etc/my.cnf

[mysqld]

server_id=1

log-bin=/data/mysql/mysql-bin

skip_name_resolve=1  #禁止反向解析

general_log    #非必须

 

mysql> show master logs;

+------------------+-----------+-----------+

| Log_name         | File_size | Encrypted |

+------------------+-----------+-----------+

| mysql-bin.000001 |       179 | No        |

| mysql-bin.000002 |       156 | No        |

+------------------+-----------+-----------+

 

#准备相关账户及权限

mysql> create user repluser@'10.0.0.%' identified by '123456';

Query OK, 0 rows affected (0.00 sec)

 

mysql> grant replication slave on *.* to repluser@'10.0.0.%';

Query OK, 0 rows affected (0.01 sec)

 

mysql> create user mhauser@'10.0.0.%' identified by '123456';

Query OK, 0 rows affected (0.01 sec)

 

mysql> grant all on *.* to mhauser@'10.0.0.%';

Query OK, 0 rows affected (0.01 sec)

 

#配置Virtual IP,在启动MHA之前未添加,启动之后master也能获取,VIP地址已在failover脚本中定义

[root@master ~]#ifconfig eth0:1 10.0.0.100/24

 

5.配置slave服务器(需在两个slave服务器上分别配置)

[root@slave1 ~]# yum -y install mysql-server

[root@slave1 ~]#mkdir /data/mysql

[root@slave1 ~]#chown mysql.mysql /data/mysql/

[root@slave1 ~]#vim /etc/my.cnf

 

[mysqld]

server_id=2   #不同节点此值不同

log-bin=/data/mysql/mysql-bin

read_only

relay_log_purge=0   #表示在默认情况下, slave 节点的中继日志会在 SQL 线程执行完毕后被自动删除,但是在 MHA 环境中,这些中继日志在恢复其它 slave 节点时可能会被用到,因此需要禁用中继日志的自动删除功能。

skip_name_resolve=1

mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.8', MASTER_USER='repluser', MASTER_PASSWORD='123456', MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=156;

mysql> start slave;

 

6.检查MHA环境

#检查环境

[root@mha-manager ~]#masterha_check_ssh --conf=/etc/mastermha/app1.cnf

[root@mha-manager ~]#masterha_check_repl --conf=/etc/mastermha/app1.cnf

#查看状态

[root@mha-manager ~]#masterha_check_status --conf=/etc/mastermha/app1.cnf

 

注意:MHA 从 0.58 版本开始增加了 super_read_only 功能,而 MariaDB-10.3.17 中没有 super_read_only 这个变量!在masterha_check_repl时报错如下,因为当时用了mysql服务器安装的是MariaDB-10.3.17,后来更换为mysql 8.0不再报错!

 

    Checking if super_read_only is defined and turned on..DBD::mysql::st execute failed: Unknown system variable 'super_read_only' at /usr/share/perl5/vendor_perl/MHA/SlaveUtil.pm line 245.

Tue Oct 20 14:42:10 2020 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln208] Slaves settings check failed!

Tue Oct 20 14:42:10 2020 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln416] Slave configuration failed.

Tue Oct 20 14:42:10 2020 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/bin/masterha_check_repl line 48.

Tue Oct 20 14:42:10 2020 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

 

7.在管理节点上启动MHA

#开启MHA,默认前台运行

[root@mha-manager ~]#nohup masterha_manager --conf=/etc/mastermha/app1.cnf &> /dev/null

[root@mha-manager ~]#masterha_manager --conf=/etc/mastermha/app1.cnf

Tue Oct 20 15:55:34 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Oct 20 15:55:34 2020 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..

Tue Oct 20 15:55:34 2020 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

 

#查看状态

[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf

app1 (pid:25452) is running(0:PING_OK), master:10.0.0.8

 

#可在master服务器上看到每秒一次的健康性检查

[root@master ~]# tail -f /var/lib/mysql/CentOS8.log

2020-10-20T08:01:00.861407Z     24 Query  SELECT 1 As Value

2020-10-20T08:01:01.861432Z     24 Query  SELECT 1 As Value

2020-10-20T08:01:02.863248Z     24 Query  SELECT 1 As Value

2020-10-20T08:01:03.863969Z     24 Query  SELECT 1 As Value

#可在master服务器上看到Virtual IP地址

[root@master ~]# ip a

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether 00:0c:29:41:a6:5d brd ff:ff:ff:ff:ff:ff

    inet 10.0.0.8/24 brd 10.0.0.255 scope global noprefixroute eth0

       valid_lft forever preferred_lft forever

    inet 10.0.0.100/8 brd 10.255.255.255 scope global eth0:1

       valid_lft forever preferred_lft forever

 

8.模拟故障

#在master服务器停止mysql服务

[root@master ~]# systemctl stop mysqld

 

#管理节点MHA自动退出

[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf

Tue Oct 20 15:55:34 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Oct 20 15:55:34 2020 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..

Tue Oct 20 15:55:34 2020 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

  Creating /data/mastermha/app1 if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /data/mysql/, up to mysql-bin.000002

Tue Oct 20 16:04:48 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Oct 20 16:04:48 2020 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..

Tue Oct 20 16:04:48 2020 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

 

[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf

app1 is stopped(2:NOT_RUNNING).

 

#排错日志显示VIP切换成功

[root@mha-manager ~]# cat /data/mastermha/app1/manager.log

----- Failover Report -----

 

app1: MySQL Master failover 10.0.0.8(10.0.0.8:3306) to 10.0.0.18(10.0.0.18:3306) succeeded

 

Master 10.0.0.8(10.0.0.8:3306) is down!

 

Check MHA Manager logs at CentOS7:/data/mastermha/app1/manager.log for details.

 

Started automated(non-interactive) failover.

Invalidated master IP address on 10.0.0.8(10.0.0.8:3306)

The latest slave 10.0.0.18(10.0.0.18:3306) has all relay logs for recovery.

Selected 10.0.0.18(10.0.0.18:3306) as a new master.

10.0.0.18(10.0.0.18:3306): OK: Applying all logs succeeded.

10.0.0.18(10.0.0.18:3306): OK: Activated master IP address.

10.0.0.28(10.0.0.28:3306): This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

10.0.0.28(10.0.0.28:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.0.0.18(10.0.0.18:3306)

10.0.0.18(10.0.0.18:3306): Resetting slave info succeeded.

Master failover to 10.0.0.18(10.0.0.18:3306) completed successfully.

Tue Oct 20 16:04:56 2020 - [info] Sending mail..

# VIP已转移至slave1服务器上

[root@slave1 ~]# ip a

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

    link/ether 00:0c:29:18:44:08 brd ff:ff:ff:ff:ff:ff

    inet 10.0.0.18/24 brd 10.0.0.255 scope global noprefixroute eth0

       valid_lft forever preferred_lft forever

    inet 10.0.0.100/8 brd 10.255.255.255 scope global eth0:1

       valid_lft forever preferred_lft forever

    inet6 fe80::20c:29ff:fe18:4408/64 scope link

       valid_lft forever preferred_lft forever

 

# slave1服务器上查看read_only已变为OFF

mysql> show global variables like 'read_only';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| read_only     | OFF   |

+---------------+-------+

 

# slave1服务器上查看slave节点为server id=3

mysql> show slave hosts;

+-----------+------+------+-----------+--------------------------------------+

| Server_id | Host | Port | Master_id | Slave_UUID                           |

+-----------+------+------+-----------+--------------------------------------+

|         3 |      | 3306 |         2 | 5ce3037d-12a8-11eb-87f1-000c29ded6af |

 

# slave2服务器上查看slave状态host地址已指向slave1的地址

mysql> show slave status\G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 10.0.0.18

                  Master_User: repluser

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000001

          Read_Master_Log_Pos: 156

               Relay_Log_File: CentOS8-relay-bin.000002

                Relay_Log_Pos: 324

        Relay_Master_Log_File: mysql-bin.000001

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

 

#查看告警邮件:

From root@CentOS7.localdomain Tue Oct 20 17:33:55 2020

Return-Path: <root@CentOS7.localdomain>

Date: Tue, 20 Oct 2020 17:33:55 +0800

To: 343882044@qq.com

Subject: MHA Warning

User-Agent: Heirloom mailx 12.5 7/5/10

Content-Type: text/plain; charset=us-ascii

From: root@CentOS7.localdomain (root)

 

MySQL is down

 

9.修复故障的master服务器

#master节点添加下面两个参数到配置文件,并启动mysqld服务

[root@master ~]# vim /etc/my.cnf

read_only

relay_log_purge=0

 

[root@master ~]# systemctl start mysqld

 

#在管理节点查看复制的起始位置

[root@mha-manager ~]# grep -i "All other slaves should start" /data/mastermha/app1/manager.log

Tue Oct 20 17:33:49 2020 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.0.18', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=156, MASTER_USER='repluser', MASTER_PASSWORD='xxx';

 

#master节点复制指向slave1(新的master)节点,并启动复制线程

mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.18', MASTER_USER='repluser', MASTER_PASSWORD='123456', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=156;

mysql> start slave;

 

#master节点查看slave状态已指向slave1(新的master)节点

mysql> show slave status\G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 10.0.0.18

                  Master_User: repluser

                  Master_Port: 3306

                Connect_Retry: 60

              Master_Log_File: mysql-bin.000001

          Read_Master_Log_Pos: 156

               Relay_Log_File: CentOS8-relay-bin.000002

                Relay_Log_Pos: 324

        Relay_Master_Log_File: mysql-bin.000001

             Slave_IO_Running: Yes

            Slave_SQL_Running: Yes

 

#在管理节点上删除 app1.failover.complete 文件

[root@mha-manager ~]# ll /data/mastermha/app1/app1.failover.complete

-rw-r--r-- 1 root root 0 Oct 20 17:33 /data/mastermha/app1/app1.failover.complete

You have mail in /var/spool/mail/root

[root@mha-manager ~]# rm -f /data/mastermha/app1/app1.failover.complete

 

#重新启动MHA服务

[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf

Tue Oct 20 18:11:36 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Tue Oct 20 18:11:36 2020 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..

Tue Oct 20 18:11:36 2020 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

 

[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf

app1 (pid:26408) is running(0:PING_OK), master:10.0.0.18

 

四、实战案例:Percona XtraDB Cluster(PXC 5.7)

 

1.准备实验环境

四台主机(centos7)

pxc1: 10.0.0.7

pxc2: 10.0.0.17

pxc3: 10.0.0.27

pxc4: 10.0.0.37

 

注意:PXC5.7目前不支持CentOS 8,主机需要关闭防火墙和SELinux并保证时间同步,如果已经安装MySQL,必须卸载

 

2,安装 Percona XtraDB Cluster 5.7

#配置yum源,使用国内yum源

[root@pxc1]# vim /etc/yum.repos.d/pxc.repo

[percona]

name=percona_repo

baseurl = https://mirrors.tuna.tsinghua.edu.cn/percona/release/$releasever/RPMS/$basearch

enabled = 1

gpgcheck = 0

 

#将yum配置文件拷贝到pxc2,pxc3上

[root@pxc1]# scp /etc/yum.repos.d/pxc.repo  10.0.0.17:/etc/yum.repos.d/

[root@pxc1]# scp /etc/yum.repos.d/pxc.repo  10.0.0.27:/etc/yum.repos.d/

 

#在3个节点都安装pxc

[root@pxc1]# yum install Percona-XtraDB-Cluster-57 -y

[root@pxc2]# yum install Percona-XtraDB-Cluster-57 -y

[root@pxc3]# yum install Percona-XtraDB-Cluster-57 -y

 

3. 在各个节点上分别配置mysql集群配置文件

/etc/my.cnf为主配置文件,当前版本中,其余的配置文件都放在/etc/percona-xtradb-cluster.conf.d目录里,包括mysqld.cnf,mysqld_safe.cnf,wsrep.cnf 三个文件

[root@pxc1 # ll /etc/percona-xtradb-cluster.conf.d/

total 12

-rw-r--r-- 1 root root  381 Oct  9 03:01 mysqld.cnf

-rw-r--r-- 1 root root  440 Oct  9 03:01 mysqld_safe.cnf

-rw-r--r-- 1 root root 1066 Oct  9 03:01 wsrep.cnf

 

#修改配置wsrep.cnf文件

[root@pxc1]# vim /etc/percona-xtradb-cluster.conf.d/wsrep.cnf

[root@pxc1]# grep -Ev "^#|^$" /etc/percona-xtradb-cluster.conf.d/wsrep.cnf

wsrep_provider=/usr/lib64/galera3/libgalera_smm.so

wsrep_cluster_address=gcomm://10.0.0.7,10.0.0.17,10.0.0.27 #添加3个节点的IP

binlog_format=ROW

default_storage_engine=InnoDB

wsrep_slave_threads= 8

wsrep_log_conflicts

innodb_autoinc_lock_mode=2

wsrep_node_address=10.0.0.7   #指定各个节点自己的IP

wsrep_cluster_name=pxc-cluster

wsrep_node_name=pxc-cluster-node-3   #指定各个节点自己的名称

pxc_strict_mode=ENFORCING

wsrep_sst_method=xtrabackup-v2

wsrep_sst_auth="sstuser:s3cretPass"   #取消注释本行,使密码验证生效

 

4. 启动PXC集群中第一个节点

#启动PXC的第一个节点

[root@pxc1 ~]# systemctl start mysql@bootstrap.service

[root@pxc1 ~]# ss -nlt

State      Recv-Q Send-Q    Local Address:Port                   Peer Address:Port             

LISTEN     0      128                   *:4567                              *:*                 

LISTEN     0      100           127.0.0.1:25                                *:*                 

LISTEN     0      128                   *:22                                *:*                 

LISTEN     0      100               [::1]:25                             [::]:*                 

LISTEN     0      80                 [::]:3306                           [::]:*                 

LISTEN     0      128                [::]:22                             [::]:*

 

#查看mysql的登录密码

[root@pxc1 ~]# grep "temporary password" /var/log/mysqld.log

2020-10-22T02:56:24.482862Z 1 [Note] A temporary password is generated for root@localhost: s%dK%rrv%5Gf

 

#登录并修改mysql的密码

[root@pxc1 ~]# mysql -uroot -p's%dK%rrv%5Gf'

mysql> alter user root@localhost identified by '123456';

#创建用于数据同步的账户并授权

mysql> create user sstuser@localhost identified by 's3cretPass';

mysql> grant reload,lock tables,process,replication client on *.* to sstuser@localhost;

 

#查看相关变量及状态

mysql> show variables like 'wsrep%'\G

mysql> show status like 'wsrep%'\G

 

Variable_name: wsrep_gcomm_uuid

        Value: 30827c2b-1412-11eb-a6ae-7a25be848952

Variable_name: wsrep_cluster_conf_id

        Value: 1

Variable_name: wsrep_cluster_size

        Value: 1

Variable_name: wsrep_cluster_status

        Value: Primary

Variable_name: wsrep_connected

        Value: ON

Variable_name: wsrep_ready

        Value: ON

Variable_name: wsrep_local_state_comment

        Value: Synced

 

 

5.启动PXC集群中其它节点

 

[root@pxc2 ~]# systemctl start mysql

[root@pxc3 ~]# systemctl start mysql

 

6. 查看集群状态,验证集群是否成功

#在任意节点,查看集群状态

[root@pxc3 ~]# mysql -uroot -p123456

 

mysql> SHOW VARIABLES LIKE 'wsrep_node_name';

+-----------------+--------------------+

| Variable_name   | Value              |

+-----------------+--------------------+

| wsrep_node_name | pxc-cluster-node-3 |

+-----------------+--------------------+

1 row in set (0.01 sec)

 

mysql> SHOW VARIABLES LIKE 'wsrep_node_address';

+--------------------+-----------+

| Variable_name      | Value     |

+--------------------+-----------+

| wsrep_node_address | 10.0.0.27 |

+--------------------+-----------+

1 row in set (0.01 sec)

 

mysql> SHOW VARIABLES LIKE 'wsrep_on';

+---------------+-------+

| Variable_name | Value |

+---------------+-------+

| wsrep_on      | ON    |

+---------------+-------+

1 row in set (0.01 sec)

 

mysql> SHOW VARIABLES LIKE 'wsrep_cluster_size';

Empty set (0.00 sec)

 

mysql> SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| wsrep_cluster_size | 3     |

+--------------------+-------+

1 row in set (0.00 sec)

 

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

+--------------------+

4 rows in set (0.00 sec)

 

#在任意节点创建数据库,并在其他节点查看数据库状态是否同步

[root@pxc3 ~]# mysql -uroot -p123456

mysql> create database testdb1;

Query OK, 1 row affected (0.01 sec)

 

[root@pxc2 ~]# mysql -uroot -p123456

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

| testdb1            |

+--------------------+

 

7. 在PXC集群中加入新节点

#在PXC集群中再加一台新的主机PXC4:10.0.0.37

[root@pxc1 ]# scp /etc/yum.repos.d/pxc.repo  10.0.0.37:/etc/yum.repos.d/

[root@pxc4 ~]# yum install Percona-XtraDB-Cluster-57 -y

#修改配置,添加IP地址到wsrep_cluster_address,其他选项做相应修改

[root@pxc4 ~]# vim /etc/percona-xtradb-cluster.conf.d/wsrep.cnf

[mysqld]

wsrep_provider=/usr/lib64/galera3/libgalera_smm.so

wsrep_cluster_address=gcomm://10.0.0.7,10.0.0.17,10.0.0.27,10.0.0.37

binlog_format=ROW

default_storage_engine=InnoDB

wsrep_slave_threads= 8

wsrep_log_conflicts

innodb_autoinc_lock_mode=2

wsrep_node_address=10.0.0.37

wsrep_cluster_name=pxc-cluster

wsrep_node_name=pxc-cluster-node-4

pxc_strict_mode=ENFORCING

wsrep_sst_method=xtrabackup-v2

wsrep_sst_auth="sstuser:s3cretPass"

#启动服务,可看到数据已同步

[root@CentOS7 ~]# systemctl start mysql

[root@CentOS7 ~]# mysql -uroot -p123456

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

| testdb1            |

+--------------------+

 

mysql>  SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| wsrep_cluster_size | 4     |

+--------------------+-------+

 

8. PXC集群中模拟节点故障并修复故障节点

 

#在任意节点停止mysql服务,这里是pxc4

[root@pxc4 ~]#systemctl stop mysql

 

#在其它任意节点查看wsrep_cluster_size变量少了一个节点

[root@pxc1 ~]#mysql -uroot -pmaged

mysql> SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| wsrep_cluster_size | 3     |

+--------------------+-------+

#创建数据库

mysql> create database testdb2;

Query OK, 1 row affected (0.00 sec)

#在其他节点上可看到数据库已同步

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

| testdb1            |

| testdb2            |

+--------------------+

 

#恢复pxc4的服务,可看到数据自动同步

[root@pxc4~]# systemctl start mysql

[root@pxc4 ~]# mysql -uroot -p123456

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sys                |

| testdb1            |

| testdb2            |

+--------------------+

7 rows in set (0.00 sec)

 

mysql>  SHOW STATUS LIKE 'wsrep_cluster_size';

+--------------------+-------+

| Variable_name      | Value |

+--------------------+-------+

| wsrep_cluster_size | 4     |

+--------------------+-------+

 

 

五、通过 ansible 部署二进制 mysql 8

 

1.环境准备

四台主机

10.0.0.18 CentOS8 做为ansible 管理端

10.0.0.8 CentOS8 远程主机

10.0.0.7 CentOS7 远程主机

10.0.0.6 CentOS6 远程主机

 

2. 安装ansible

#使用yum安装

[root@Ansible ~]# yum -y install ansible

[root@Ansible ~]# ll /etc/ansible/

total 24

-rw-r--r-- 1 root root 19985 Oct  8 02:06 ansible.cfg

-rw-r--r-- 1 root root  1016 Oct  8 02:06 hosts

drwxr-xr-x 2 root root     6 Oct  8 02:06 roles

 

3. 添加host

[root@Ansible ~]# vim /etc/ansible/hosts

[dbservers]

10.0.0.8

10.0.0.7

 

[webservers]

10.0.0.18 ansible_connection=local

10.0.0.6

 

[appservers]

10.0.0.8

10.0.0.6

 

[root@Ansible ~]# ansible all  --list-hosts

  hosts (4):

    10.0.0.8

    10.0.0.7

    10.0.0.18

    10.0.0.6

 

4.实现基于key验证的远程主机登录

#准备基于key验证的脚本

[root@Ansible ~]# vim ssh_push_key.sh

IPLIST="

10.0.0.8

10.0.0.18

10.0.0.7

10.0.0.6

"

rpm -q sshpass &> /dev/null || yum -y install sshpass 

[ -f /root/.ssh/id_rsa ] || ssh-keygen -f /root/.ssh/id_rsa  -P ''

export SSHPASS=centos

for IP in $IPLIST;do

    sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $IP

done

 

#运行脚本,实现基于key验证的远程主机登录

[root@Ansible ~]# bash ssh_push_key.sh

 

#使用ping验证

[root@Ansible ~]# ansible all -m ping

10.0.0.18 | SUCCESS => {

    "ansible_facts": {

        "discovered_interpreter_python": "/usr/libexec/platform-python"

    },

    "changed": false,

    "ping": "pong"

}

10.0.0.8 | SUCCESS => {

    "ansible_facts": {

        "discovered_interpreter_python": "/usr/libexec/platform-python"

    },

    "changed": false,

    "ping": "pong"

}

10.0.0.7 | SUCCESS => {

    "ansible_facts": {

        "discovered_interpreter_python": "/usr/bin/python"

    },

    "changed": false,

    "ping": "pong"

}

10.0.0.6 | SUCCESS => {

    "ansible_facts": {

        "discovered_interpreter_python": "/usr/bin/python"

    },

    "changed": false,

    "ping": "pong"

}

 

5.准备相关文件和包

[root@Ansible ~]# mkdir -pv /data/ansible/files

#准备mysql 8.0的安装包,上传到/data/ansible/files/下

#准备配置文件

[root@Ansible ~]# vim /data/ansible/files/my.cnf

[mysqld]

user=mysql

log-bin

symbolic-links=0

datadir=/data/mysql

socket=/data/mysql/mysql.sock

innodb_file_per_table=1

pid-file=/data/mysql/mysqld.pid

log-error=/data/mysql/mysql.log

 

[client]

port=3306

socket=/data/mysql/mysql.sock

 

[mysqld_safe]

log-error=/var/log/mysqld.log

 

#准备mysql安装脚本

[root@Ansible ~]# vim /data/ansible/install_mysql.yml

---

# install mysql-8.0.19-linux-glibc2.12-x86_64.tar.xz

- hosts: [dbservers]

  remote_user: root

  gather_facts: no

 

  tasks:

    - name: install packages

      yum: name=libaio,autoconf,libtinfo*,libncurses*,perl-Data-Dumper,numactl-libs

    - name: create mysql group

      group: name=mysql gid=306

    - name: create mysql user

      user: name=mysql uid=306 group=mysql shell=/sbin/nologin system=yes

    - name: copy tar to remote host and file mode

      unarchive: src=/data/ansible/files/mysql-8.0.19-linux-glibc2.12-x86_64.tar.xz dest=/usr/local owner=root group=root

    - name: create linkfile /usr/local/mysql

      file: src=/usr/local/mysql-8.0.19-linux-glibc2.12-x86_64 dest=/usr/local/mysql state=link

    - name: PATH variable

      copy: content='PATH=/usr/local/mysql/bin:$PATH' dest=/etc/profile.d/mysql.sh

    - name: PATH variable source

      shell: source /etc/profile.d/mysql.sh

    - name: config my.cnf

      copy: src=/data/ansible/files/my.cnf  dest=/etc/my.cnf

    - name: data dir

      shell: mysqld --initialize --user=mysql --datadir=/data/mysql

    - name: service script

      shell: /bin/cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld

    - name: enable service

      shell: /etc/init.d/mysqld start;chkconfig --add mysqld;chkconfig mysqld on

    - name: change mysql password

      shell: mysqladmin  -uroot -p`awk '/A temporary password/{print $NF}' /data/mysql/mysql.log` password 123456

 

6. 开始安装mysql8

[root@Ansible ~]# ansible-playbook /data/ansible/install_mysql.yml

 

PLAY [10.0.0.8] *******************************************

 

TASK [install packages] **********************************

changed: [10.0.0.8]

 

TASK [create mysql group] ****************************

changed: [10.0.0.8]

 

TASK [create mysql user] ********************************

changed: [10.0.0.8]

 

TASK [copy tar to remote host and file mode] *************

changed: [10.0.0.8]

 

TASK [create linkfile /usr/local/mysql] ******************

changed: [10.0.0.8]

 

TASK [PATH variable] **************************************

changed: [10.0.0.8]

 

TASK [PATH variable source] ******************************

changed: [10.0.0.8]

 

TASK [config my.cnf] *************************************

changed: [10.0.0.8]

 

TASK [data dir] ******************************************

changed: [10.0.0.8]

 

TASK [service script] *************************************

changed: [10.0.0.8]

 

TASK [enable service] *************************************

changed: [10.0.0.8]

 

TASK [change mysql password] *****************************

changed: [10.0.0.8]

 

PLAY RECAP *************************************************

10.0.0.8  : ok=12   changed=12   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

#在远程服务器上登录验证

[root@CentOS8 ~]# mysql -uroot -p123456

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 10

Server version: 8.0.19 MySQL Community Server - GPL

 

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

 

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

 

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 

mysql>

 

【软件工程B】2021-2022(2)作业【五、六】 【软件工程B】2021-2022(2)作业【五、六】 总评A+,答案仅供参考 作业五 绘制如下两个系统的功能层次结构图(至少3层,可以合理地虚拟功能,可打印粘贴) 学生基础信息管理系统 手机操作系统 作业六 绘制一个程序流程图(框图): 判断一个学号的前4位和后4位是否规范(假设一个专业最多4个班、一个班最多45个人) (本题题干中并未说明专业班级数和班级人数与学号前后四位之间的关系,显得有些莫名其妙,是否漏掉什么重要信息,望指正) ... 阅读详情

相关推荐

ROS 2 Slam Toolbox实战:从零跑通2D激光建图

SLAM(同步定位与地图构建)是移动机器人实现自主导航的基础技术,其核心在于通过传感器数据实时估计位姿并构建环境栅格地图。slam_toolbox作为ROS 2 Humble及后续LTS版本官方主推的2D SLAM后端,采用图优化+闭环检测架构,兼顾实时性、鲁棒性与工程可维护性。相比cartographer和hector_slam,它原生支持ROS 2、参数精简、诊断友好,特别适合硬件资源有限、缺乏IMU或深度相机的差速机器人平台。本文聚焦真实部署场景,覆盖RPLIDAR等常见2D激光雷达与TurtleBo

weixin_30596165的博客 358

Mysql高可用-----MHA

一、 MHA简介 MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Facebook公司)开发,是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中,MHA能做到在0~30秒之内自动完成数据库的故障切换操作,并且在进行故障切换的过程中,MHA...

菜鸟、上路 692

在 Ansys CFX Pre 中配置 RGP 表的分步指南

掌握在 Ansys CFX Pre 中设置 RGP 表的技巧,以优化仿真精度和效率。

jkh920184196的博客 1671

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failove

[root@192 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING). [1]+ Exit 1 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/l...

oldbob2020的博客 958

LSF实践专题(15):LSF队列优先级以及抢占功能初探

在LSF暂停作业时,会向作业发送一个SIGTSTP信号,并且将作业状态变成SSUSP(system suspend),表示被系统暂停,这时,最好是作业的应用能够捕获和处理SIGTSTP信号,否则,即使在LSF中,作业会被认为暂停了,并且释放了slots资源,但是在实际OS中,作业进程仍然在继续,并且不会释放CPU等计算资源。是指LSF自动将低优先级作业暂停(suspend),将空余出来的资源交给高优先级队列中的作业使用,等到高优先级队列的作业结束后,再将被暂停的作业恢复(resume)并继续运行。

richycn的博客 1920

网络流量监测技术

NetFlow是Cisco公司提出的网络数据包交换技术,该技术首先被用于网络设备对数据交换进行加速,并可同步实现对高速转发的IP数据流(Flow)进行测量和统计。  随着宽带互联网在中国 的迅速发展,全国各大电信运营商的网络规模都在不断扩张,网络结构日渐复杂,网络业务日趋丰富,网络流量高速增长。电信运营商需要通过可靠、有效的网络业 务流量监测系统对其网络以及网络所承载的各类业务进行及时、准

红杉树(ecitnet)的博客 5541

mysql查询表时报1142_mysql-MHA切换或检查复制时报1142问题定位和解决

====模拟当前slaves之间relay log存在差异,当前master mysql服务关闭后的mha自动切换====1、在没有设置master_ip_failover_script的情况下的切换测试:测试预想结果:自动切换成功=============================================1)在241上,制造relay log同步不一致情况mysql> sto...

weixin_39653761的博客 652

基于深度学习的道路裂缝检测系统完整教程

本文介绍了一种基于深度学习的道路裂缝检测系统完整解决方案。项目采用5466张高质量道路裂缝图像数据集,包含YOLO和Pascal VOC两种标注格式,覆盖多种道路场景。系统利用CNN和YOLO算法实现高效检测,相比传统人工巡检显著提升效率和准确性。文中详细阐述了数据集结构、环境配置和训练流程,包括数据预处理、智能分割策略和质量检查方法。该技术可及时发现道路裂缝隐患,降低维护成本,保障交通安全,为道路养护提供智能化解决方案。

FJN110的博客 298

mysql-MHA 故障收集

在manager 主机上开启监控服务,启动不了 [root@manager ~]# managerStart [1] 1472 [root@manager ~]# managerStatus app1 is stopped(2:NOT_RUNNING). [1]+ Exit 1 nohup masterha_manager --conf=/etc/...

weixin_30681615的博客 953

Python实战:打造学生信息管理系统

通过本教程,我们构建了一个简单的学生信息管理系统,实现了基本的添加、删除、修改和查询功能。当然,这个系统还有很多可以改进和扩展的地方,比如添加用户界面、增加数据持久化功能、支持批量操作等。简介:本文将介绍如何使用Python编写一个功能齐全的学生信息管理系统,包括添加、删除、修改和查询学生信息的基本操作。学生信息管理系统是学校日常管理中不可或缺的一部分,它能够帮助管理人员快速、准确地管理学生信息,提高工作效率。为了实现上述功能,我们可以设计一个包含学生信息的字典,并使用函数来封装各个操作。

铭渊老黄 1492

ROS机器人未知环境自主探索功能包explore_lite最全源码详细解析(二)

本系列文章主要针对ROS机器人常使用的未知环境自主探索功能包explore_lite展开全源码的详细解析,并进行概括总结。 本系列文章共包含六篇文章,前五篇文章主要介绍explore_lite功能包中 explore.cpp、costmap_tools.h、frontier_search.cpp、costmap_client.cpp等源码实现文件中全部函数的详细介绍,第六篇文章进行概括总结,包含自主探索功能包explore_lite的简介,实现未知环境自主探索功能的原理及流程总结,效果演示,使用功能包exp

慕羽★的博客 1648

作业五思维导图

2201_75562774的博客 183

《派出你的AI同事:WorkBuddy案例实战》055:客服工单自动分类与话术生成

客服工单自动分类与话术生成 —— 《派出你的AI同事:WorkBuddy案例实战》系列文章,配图已统一为规范示意图。

LuckyHanMo的博客 285

课程作业五

课程作业五 题目描述 ·针对第四次作业的完善。请根据第四次作业分析设计的结果进行编码。要求根据设计的类图进行编码,写出实现该程序所需要的类。 ·学习数据结构栈的知识。 链接 github 数据结构栈的学习 基本概念 栈 (Stack)是一种后进先出(last in first off,LIFO)的数据结构。 栈的实现 1.Stack<T>()创建一个空的栈 2.void P...

ayeql82242的专栏 162

pscad IEEE标准系统搭建说明

pscad IEEE标准系统搭建说明,包含典型的IEEE 9节点、14节点、30节点、39节点和118节点系统。内含标准测试系统的原始数据和潮流结果。

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

操作系统作业五

操作系统作业五

weixin_50304297的博客 348

汉书HE8,好用的汇编代码编写工具

汉书HE8,好用的汇编代码编写工具早期编写汇编代码所用功能强大

wp的作业五

查找学生信息,首先要对学生信息进行录入,再遍历学生数组,利用strcmp函数比较学生姓名是否与需要查找的学生姓名相同,如果相同就输出学生的所有信息,如果数组中没有信息与之相同返回Not Found提示语句。学生根据平均分排序的时候,首先需要获取平均分,平均分获取方法和上题相同,排序的时候是对数组内的元素进行排序,因为数组的数据类型地址值不变,所以执行完函数之后数组的排序仍旧是函数内的排序。此处需要注意平均分的小数,最后需要进行强转,因为定义的时候是整数类型。

2405_87353547的博客 587

NTscan扫描工具

ntscan是一个扫描器,可以扫描没有密码和弱密码的共享,先到配置文件中填写猜测的用户名和密码,扫描过程中右侧出现的是扫描日志。

HNU数据结构与算法-作业五

因为时间问题,不提供思路。

bz_czq的博客 488

ComSec作业五:椭圆曲线

ComSec作业五:椭圆曲线

m0_52190697的博客 160

数据库作业五

【代码】数据库作业五。

STEVEHlyq的博客 140

2017-2018-1 JAVA实验站 第四、五周作业

2017-2018-1 JAVA实验站 第四、五周作业 JAVA实验站 第四、五周作业 转载于:https://www.cnblogs.com/Tiffany23/p/7788724.html

bafeiqiao7632的博客 116

五、 作业

1. 想结束一个进程及其子进程在windows下很难,因为windows没有维护进程间的父子关系 2. windows 提供一个作业内核对象,它允许我们将进程组合在一起并创建一个“沙箱”, 来限制进程能做什么 3. 只有在作业中的所有进程都已终止运行之后,才会自动销毁 4. 作业的cpu时间使用完,会杀死作业的所有进程 待细化。。

vsooda的专栏 911

GAMES101:现代计算机图形学入门-作业五

这次作业给了许多脚本,我们现在可以把每个脚本的代码逐行细细分析一下。

lastXuanGod的博客 1144

CINTA作业五:循环群

why的博客 121

系统分析与设计作业五

系统分析与设计作业五 文章目录系统分析与设计作业五1、根据订旅馆建模文档2、根据课程练习“投递员使用投递箱给收件人快递包裹”的业务场景 使用 UMLet 建模: 1、根据订旅馆建模文档 ​ Asg-RH.pdf 绘制用例图模型(到子用例) 给出 make reservation 用例的活动图 2、根据课程练习“投递员使用投递箱给收件人快递包裹”的业务场景 分别用多泳道图建模三个场景...

qq_31379559的博客 210

五:作业

Windows 提供了一个Job内核对象,它

艰难困苦如同欢乐,终将成为人生最后的财富 564
上一篇: 作业四
下一篇: 作业六
Learning_Linux
博客等级 码龄6年 0粉丝 9原创
评论 1
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值