本帖最后由 飞鸿无痕 于 2010-11-11 23:04 编辑 mysql-mmm+amoeba+keepalived架构简介:利用mysql-mmm来实现mysql的高可用,利用amoeba来实现读写分离,利用keepalived来实现amoeba程序的高可用。从而从整体上实现mysql的高可用行。一、 MMM简介:MMM即Master-Master Replication Manager for MySQL(mysql主主复制管理器)关于mysql主主复制配置的监控、故障转移和管理的一套可伸缩的脚本套件(在任何时候只有一个节点可以被写入),这个套件也能对居于标准的主从配置的任意数量的从服务器进行读负载均衡,所以你可以用它来在一组居于复制的服务器启动虚拟ip,除此之外,它还有实现数据备份、节点之间重新同步功能的脚本。MySQL本身没有提供replication failover的解决方案,通过MMM方案能实现服务器的故障转移,从而实现mysql的高可用。MMM不仅能提供浮动IP的功能,更牛的是如果当前的主服务器挂掉后,会将你后端的从自动转向新的主服务器进行同步复制,不用手工更改同步资料。MMM项目来自 Google:http://code.google.com/p/mysql-master-master官方网站为:http://mysql-mmm.orgMmm主要功能由下面三个脚本提供 mmm_mond 负责所有的监控工作的监控守护进程,决定节点的移除等等 mmm_agentd 运行在mysql服务器上的代理守护进程,通过简单远程服务集提供给监控节点 mmm_control 通过命令行管理mmm_mond进程二、 mysql-mmm架构的搭建1、 先来看下本文的架构图:当master1出现故障的时候,架构图会自动屏蔽master1,并自动将slave1和slave2更改成从master2更新,架构图变成如下:2、 先介绍下本文的环境:系统环境:CentOS release 5.4(32bit)function ip hostname server id Monitor 192.168.1.163 Server3 - master 1 192.168.1.161 Server1 1 master 2 192.168.1.162 Server2 2 slave 1 192.168.1.164 Server4 5 slave 2 192.168.1.165 Server5 6 Amoeba1 192.168.1.167 Server6 -Amoeba2 192.168.1.168 Server7 -我用了以下的虚拟IP,他们将会在被mmm分配,这些IP都是浮动的,当出现故障的时候会自动的转移到其他的可用服务器上。ip role description192.168.1.113 writer 你的应用程序应该连接到这个ip进行写操作192.168.1.111 reader 你的应用程序应该链接到这些ip中的一个进行读操作192.168.1.112 reader 192.168.1.114 reader 192.168.1.115 reader 192.168.1.170 Amoeba_vip 程序连接的Ip,实现mysql读写分离mysql-mmm架构配置简介: 在server1、server2上安装mysql,并配置为master-master架构(就是互为主从)----------配置很简单,就不对着部分进行详细解释,有问题的话请查看:http://blog.chinaunix.net/u3/93755/showart.php?id=2213538 配置server4,server5从server1获取更新 ------这一步很简单,自行设置,有一点需要注意,就是设置同步不能设置浮动IP,要使用真实的IP。 在server1、server2、server3、server4、server5上安装mmm,并配置:mmm_common.conf、mmm_agent.conf以及mmm_mon.conf文件3、 Mysql-mmm实战前提:server1和server2上已经配置好mysql主主同步,server4、server5设置好了利用server1为主服务器的主从同步 安装mysql-mmm(在server1,server2,server3,server4,server5上都要安装)CentOS软件仓库默认是不含这些软件的,必须要有epel这个包的支持。故我们必须先安装epel: wget http://download.fedora.redhat.co ... ease-5-4.noarch.rpmrpm -Uvh epel-release-5-4.noarch.rpmyum -y install mysql-mmm* 复制代码 配置mmm代理和监控账号的权限在server1,server2,server4,server5上分别执行: GRANT REPLICATION CLIENT ON *.* TO 'mmm_monitor'@'192.168.1.%' IDENTIFIED BY 'monitor_password';GRANT SUPER, REPLICATION CLIENT, PROCESS ON *.* TO 'mmm_agent'@'192.168.1.%' IDENTIFIED BY 'agent_password';flush privileges; 复制代码 配置mysql-mmm所有的配置选项都集合在了一个叫/etc/mysql-mmm/mmm_common.conf的单独文件中,系统中所有主机的该文件内容都是一样的, 配置完后不要忘记了拷贝这个文件到所有的主机(包括监控主机)!,内容如下: active_master_role writer<host default> cluster_interface eth0 pid_path /var/run/mysql-mmm/mmm_agentd.pidbin_path /usr/libexec/mysql-mmm/#同步的帐号(这些要和前面设置的保持一致!) replication_user replication replication_password 123456 #同步的密码 agent_user mmm_agent #mmm-agent用户名 agent_password agent_password #mmm-agent用户密码</host><host db1> ip 192.168.1.161 #db1的ip mode master peer db2</host><host db2> ip 192.168.1.162 #db2的ip mode master peer db1</host><host db3> ip 192.168.1.164 #从db3的ip mode slave</host><host db4> ip 192.168.1.165 #从db4的ip mode slave</host><role writer> hosts db1, db2 ips 192.168.1.113 #设置写如的虚拟IP mode exclusive</role><role reader> hosts db1, db2, db3, db4 ips 192.168.1.111, 192.168.1.112, 192.168.1.114, 192.168.1.115 #设置读取的虚拟IP mode balanced</role> 复制代码 在数据库主机上我们需要编辑/etc/mysql-mmm/mmm_agent.conf文件,根据其他主机的不同更改db1的值(db2就将db1更改成db2…….): include mmm_common.confthis db1 复制代码 在监控主机上我们需要编辑/etc/mysql-mmm/mmm_mon.conf文件: include mmm_common.conf<monitor> ip 127.0.0.1 pid_path /var/run/mysql-mmm/mmm_mond.pid bin_path /usr/libexec/mysql-mmm status_path /var/lib/mysql-mmm/mmm_mond.status ping_ips 192.168.1.161,192.168.1.162, 192.168.1.164,192.168.1.165 #监控服务器ip auto_set_online 60 # The kill_host_bin does not exist by default, though the monitor will # throw a warning about it missing. See the section 5.10 "Kill Host # Functionality" in the PDF documentation. # # kill_host_bin /usr/libexec/mysql-mmm/monitor/kill_host #</monitor><host default> monitor_user mmm_monitor #mmm_monitor用户名 monitor_password monitor_password #mmm_monitor密码</host>debug 0 复制代码 启动MMM启动代理:(在数据库服务器上server1、2)编辑/etc/default/mysql-mmm-agent来开启: ENABLED=1 复制代码 然后启动它: /etc/init.d/mysql-mmm-agent start 复制代码 启动监控(在监控机上): /etc/init.d/mysql-mmm-monitor start 复制代码 利用mmm_control监控mysql服务器状态: [root@server3 ~]# mmm_control show db1(192.168.1.161) master/ONLINE. Roles: reader(192.168.1.111), writer(192.168.1.113) db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.114) db3(192.168.1.164) slave/ONLINE. Roles: reader(192.168.1.115) db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.112) 复制代码 测试看两个mysql服务器能否实现故障自动切换停掉作为写的db1上的mysql,查看写的服务器会不会自动转移到db2上去停掉几秒钟后用mmm_control show查看: [root@server3 ~]# mmm_control show db1(192.168.1.161) master/HARD_OFFLINE. Roles: db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113) db3(192.168.1.164) slave/ONLINE. Roles: reader(192.168.1.114), reader(192.168.1.115) db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111) 复制代码 我们可以看到已经把db2当作主写服务器,另外server4、server5原来是从server1更新的现在已经被mmm自动更改到从新的主服务器server2上更新了,很神奇!可以登录到server4、server5上用show slave status\G;命令查看!再来看看db1恢复后会是什么情况: [root@server3 ~]# mmm_control show db1(192.168.1.161) master/ONLINE. Roles: reader(192.168.1.114) db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113) db3(192.168.1.164) slave/ONLINE. Roles: reader(192.168.1.115) db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111) 复制代码 我们可以看到当db1恢复后就充当slave的角色了!只有当db2挂了以后db1又会担当起主服务器的写入功能发现个bug,不知道有没有人遇到过:有的时候会出现服务器恢复后,服务器的状态被设置为:AWAITING_RECOVERY,必须要手工执行mmm_control set_online host才能恢复正常,比如:有的时候我恢复db1后,出现如下: [root@server3 ~]# mmm_control show db1(192.168.1.161) master/AWAITING_RECOVERY. Roles: db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113) db3(192.168.1.164) slave/ONLINE. Roles: reader(192.168.1.114), reader(192.168.1.115) db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111) 复制代码 解决办法为利用set_online手工更改为在线状态: [root@server3 ~]# mmm_control set_online db1OK: State of 'db1' changed to ONLINE. Now you can wait some time and check its new roles![root@server3 ~]# mmm_control show db1(192.168.1.161) master/ONLINE. Roles: reader(192.168.1.114) db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113) db3(192.168.1.164) slave/ONLINE. Roles: reader(192.168.1.115) db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111) 复制代码 解决这个问题CU网友提供了一个办法就是将auto_set_online设置小点,默认是60s,设置成10看下会不会有改善?我测试的是改小后没出现,回头多测试几次看看。 mmm_control命令简介 [root@server3 mysql-mmm]# mmm_control helpValid commands are: help - show this message #查看帮助信息ping - ping monitor#ping监控show - show status#查看状态信息checks [<host>|all [<check>|all]] - show checks status#显示检查状态,包括(ping、mysql、rep_threads、rep_backlog)set_online <host> - set host <host> online#设置某host为online状态set_offline <host> - set host <host> offline#设置某host为offline状态mode - print current mode.#打印当前的模式,是ACTIVE、MANUAL、PASSIVE?#默认ACTIVE模式set_active - switch into active mode.#更改为active模式set_manual - switch into manual mode.#更改为manual模式set_passive - switch into passive mode.#更改为passive模式move_role [--force] <role> <host> - move exclusive role <role> to host <host>#更改host的模式,比如更改处于slave的mysql数据库角色为write (Only use --force if you know what you are doing!)set_ip <ip> <host> - set role with ip <ip> to host <host>#为host设置ip,只有passive模式的时候才允许更改! 三、 利用amoeba实现高可用下的读写分离 1、 关于amoeba的介绍和安装我这里不做介绍了,请查看:http://blog.chinaunix.net/u3/93755/showart.php?id=2363202,有什么问题我们再一起讨论。 2、 Amoeba在mysql-mmm架构下的配置配置的除了amoeba的监听IP外其他的均采用上面mmm分配的浮动IP,因为当出现故障后,浮动IP会自动分配到正常的server上。amoeba.xml的配置如下: <?xml version="1.0" encoding="gbk"?><!DOCTYPE amoeba:configuration SYSTEM "amoeba.dtd"><amoeba:configuration xmlns:amoeba="http://amoeba.meidusa.com/"> <server> <!-- proxy server绑定的端口 --> <property name="port">3306</property> <!-- proxy server绑定的IP要设置成0.0.0.0,因为后面还要监听VIP --> <property name="ipAddress">0.0.0.0</property> <!-- proxy server net IO Read thread size --> <property name="readThreadPoolSize">20</property> <!-- proxy server client process thread size --> <property name="clientSideThreadPoolSize">30</property> <!-- mysql server data packet process thread size --> <property name="serverSideThreadPoolSize">30</property> <!-- socket Send and receive BufferSize(unit:K) --> <property name="netBufferSize">128</property> <!-- Enable/disable TCP_NODELAY (disable/enable Nagle's algorithm). --> <property name="tcpNoDelay">true</property> <property name="user">root</property> <property name="password">password</property> <!-- query timeout( default: 60 second , TimeUnit:second) --> <property name="queryTimeout">60</property> </server> <!-- 每个ConnectionManager都将作为一个线程启动。 manager负责Connection IO读写/死亡检测 --> <connectionManagerList> <connectionManager name="defaultManager" class="com.meidusa.amoeba.net.MultiConnectionManagerWrapper"> <property name="subManagerClassName">com.meidusa.amoeba.net.AuthingableConnectionManager</property> <!-- default value is avaliable Processors <property name="processors">5</property> --> </connectionManager> </connectionManagerList> <dbServerList> <!-- 一台mysqlServer 需要配置一个pool, 如果多台 平等的mysql需要进行loadBalance, 平台已经提供一个具有负载均衡能力的objectPool:com.meidusa.amoeba.mysql.server.MultipleServerPool 简单的配置是属性加上 virtual="true",该Pool 不允许配置factoryConfig 或者自己写一个ObjectPool。 --> <dbServer name="server1"> <!-- PoolableObjectFactory实现类 --> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">defaultManager</property> <!-- 真实mysql数据库端口 --> <property name="port">3306</property> <!-- 真实mysql数据库IP --> <property name="ipAddress">192.168.1.113</property> <property name="schema">test</property> <!-- 用于登陆mysql的用户名 --> <property name="user">zhang</property> <!-- 用于登陆mysql的密码 --> <property name="password">zhang123</property> </factoryConfig> <!-- ObjectPool实现类 --> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">200</property> <property name="maxIdle">200</property> <property name="minIdle">10</property> <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="server2"> <!-- PoolableObjectFactory实现类 --> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">defaultManager</property> <!-- 真实mysql数据库端口 --> <property name="port">3306</property> <!-- 真实mysql数据库IP --> <property name="ipAddress">192.168.1.111</property> <property name="schema">test</property> <!-- 用于登陆mysql的用户名 --> <property name="user">zhang</property> <!-- 用于登陆mysql的密码 --> <property name="password">zhang123</property> </factoryConfig> <!-- ObjectPool实现类 --> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">200</property> <property name="maxIdle">200</property> <property name="minIdle">10</property> <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="server3"> <!-- PoolableObjectFactory实现类 --> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">defaultManager</property> <!-- 真实mysql数据库端口 --> <property name="port">3306</property> <!-- 真实mysql数据库IP --> <property name="ipAddress">192.168.1.112</property> <property name="schema">test</property> <!-- 用于登陆mysql的用户名 --> <property name="user">zhang</property> <!-- 用于登陆mysql的密码 --> <property name="password">zhang123</property> </factoryConfig> <!-- ObjectPool实现类 --> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">200</property> <property name="maxIdle">200</property> <property name="minIdle">10</property> <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="server4"> <!-- PoolableObjectFactory实现类 --> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">defaultManager</property> <!-- 真实mysql数据库端口 --> <property name="port">3306</property> <!-- 真实mysql数据库IP --> <property name="ipAddress">192.168.1.114</property> <property name="schema">test</property> <!-- 用于登陆mysql的用户名 --> <property name="user">zhang</property> <!-- 用于登陆mysql的密码 --> <property name="password">zhang123</property> </factoryConfig> <!-- ObjectPool实现类 --> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">200</property> <property name="maxIdle">200</property> <property name="minIdle">10</property> <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="server5"> <!-- PoolableObjectFactory实现类 --> <factoryConfig class="com.meidusa.amoeba.mysql.net.MysqlServerConnectionFactory"> <property name="manager">defaultManager</property> <!-- 真实mysql数据库端口 --> <property name="port">3306</property> <!-- 真实mysql数据库IP --> <property name="ipAddress">192.168.1.115</property> <property name="schema">test</property> <!-- 用于登陆mysql的用户名 --> <property name="user">zhang</property> <!-- 用于登陆mysql的密码 --> <property name="password">zhang123</property> </factoryConfig> <!-- ObjectPool实现类 --> <poolConfig class="com.meidusa.amoeba.net.poolable.PoolableObjectPool"> <property name="maxActive">200</property> <property name="maxIdle">200</property> <property name="minIdle">10</property> <property name="minEvictableIdleTimeMillis">600000</property> <property name="timeBetweenEvictionRunsMillis">600000</property> <property name="testOnBorrow">true</property> <property name="testWhileIdle">true</property> </poolConfig> </dbServer> <dbServer name="master" virtual="true"> <poolConfig class="com.meidusa.amoeba.server.MultipleServerPool"> <!-- 负载均衡参数 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA--> <property name="loadbalance">1</property> <!-- 参与该pool负载均衡的poolName列表以逗号分割 --> <property name="poolNames">server1</property> </poolConfig> </dbServer> <dbServer name="slave" virtual="true"> <poolConfig class="com.meidusa.amoeba.server.MultipleServerPool"> <!-- 负载均衡参数 1=ROUNDROBIN , 2=WEIGHTBASED , 3=HA--> <property name="loadbalance">1</property> <!-- 参与该pool负载均衡的poolName列表以逗号分割 --> <property name="poolNames">server2,server3,server4,server5</property> </poolConfig> </dbServer> </dbServerList> <queryRouter class="com.meidusa.amoeba.mysql.parser.MysqlQueryRouter"> <property name="ruleConfig">${amoeba.home}/conf/rule.xml</property> <property name="functionConfig">${amoeba.home}/conf/functionMap.xml</property> <property name="ruleFunctionConfig">${amoeba.home}/conf/ruleFunctionMap.xml</property> <property name="LRUMapSize">1500</property> <property name="defaultPool">master</property> <property name="writePool">master</property> <property name="readPool">slave</property> <property name="needParse">true</property> </queryRouter></amoeba:configuration> 3、 启动amoeba /usr/local/amoeba/bin/amoeba & 检验启动是否成功(使用的是默认的8066端口): [root@server3 ~]# ps aux | grep amoebaroot 15390 0.2 21.7 411800 55684 pts/0 Sl Nov09 0:13 /usr/java/jdk1.6/bin/java -server -Xms256m -Xmx256m -Xss128k -Damoeba.home=/usr/local/amoeba -Dclassworlds.conf=/usr/local/amoeba/bin/amoeba.classworlds -classpath /usr/local/amoeba/lib/classworlds-1.0.jar org.codehaus.classworlds.Launcher 4、 测试为了有权限,请在所有mysql-server上添加上面配置的用户名和密码,比如: grant all on test.* to zhang@'192.168.1.%' identified by 'zhang123';flush privileges; 测试的时候比较麻烦,因为如果把主从停掉来看效果的话,mmm会将该同步状态有问题的服务器设置成REPLICATION_FAIL并将该主机上的浮动IP移动到正常服务器,比如:我将server4停掉后,就出出现如下现象: [root@server3 ~]# mmm_control show db1(192.168.1.161) master/ONLINE. Roles: reader(192.168.1.114), reader(192.168.1.115) db2(192.168.1.162) master/ONLINE. Roles: reader(192.168.1.112), writer(192.168.1.113) db3(192.168.1.164) slave/REPLICATION_FAIL. Roles: db4(192.168.1.165) slave/ONLINE. Roles: reader(192.168.1.111) 暂时没想到很好的测试办法。读写分离测试没有问题。最近手头没有空闲的机器来做压力测试。等有了机器了再对这个架构做下压力测试!四、 利用keepalived实现amoeba高可用Keepalived简介:Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接。从而提高系统的可用性1、 keepalived的安装(server6、server7上都要安装): wget [url]http://www.keepalived.org/software/keepalived-1.1.19.tar.gz[/url]tar zxvf keepalived-1.1.19.tar.gzcd keepalived-1.1.19./configure --prefix=/usr/local/keepalivedmake make installcp /usr/local/keepalived/sbin/keepalived /usr/sbin/cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/mkdir /etc/keepalivedcd /etc/keepalived/ 2、 keepalived的配置 配置server6的keepalived配置文件/etc/keepalived/keepalived.conf,内容如下: bal_defs { notification_email { [email]jimo291@gmail.com[/email] } notification_email_from [email]jimo291@gmail.com[/email] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id test1}vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 smtp_alert authentication { auth_type PASS auth_pass 123 } virtual_ipaddress { 192.168.1.170 }} 配置server6的keepalived配置文件/etc/keepalived/keepalived.conf,内容如下: bal_defs { notification_email { [email]jimo291@gmail.com[/email] } notification_email_from [email]jimo291@gmail.com[/email] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id test2}vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 80 advert_int 1 smtp_alert authentication { auth_type PASS auth_pass 123 } virtual_ipaddress { 192.168.1.170 }} 3、 keepalived的启动: 在server6、server7上执行如下命令启动: /etc/rc.d/init.d/keepalived start 看看server6上是否绑定了虚拟IP192.168.1.170 [root@server6 ~]# ip add | grep 192.168.1.170inet 192.168.1.170/32 scope global eth0 可以看到已经成功绑定了192.168.1.170。4、 测试停掉server6,看看虚拟IP192.168.1.170会不会自动切换到server7上。以及测试mysql -uroot -ppassword -h192.168.1.170登录,看看能否实现读写操作是否正常等等!备注:最后记得将所有的启动命令都写到/etc/rc.local文件中去,实现开机启动!