准备8台redis服务器
名称 | ip | 端口 |
---|---|---|
redisA | 192.168.8.51 | 6351 |
redisB | 192.168.8.52 | 6352 |
redisC | 192.168.8.53 | 6353 |
redisD | 192.168.8.54 | 6354 |
redisE | 192.168.8.55 | 6355 |
redisF | 192.168.8.56 | 6356 |
redisG | 192.168.8.57 | 6357 |
redisH | 192.168.8.58 | 6358 |
1、部署redis集群
(1)环境准备
[root@host51 ~]# redis-cli -h 192.168.8.51 -p 6358 shutdown
[root@host51 ~]# vim /etc/redis/6379.conf
70 bind 192.168.8.51 //接收连接请求的ip地址
93 port 6351 //redis监听端口
815 cluster-enabled yes //开启集群模式
823 cluster-config-file nodes-6351.conf //保存集群节点信息文件
829 cluster-node-timeout 5000 //集群节点超时响应时间5000ms
[root@host51 ~]# /etc/init.d/redis_6379 start
Starting Redis server...
[root@host51 ~]# netstat -ntulp |grep 6351 //集群监听端口16351
tcp 0 0 192.168.8.51:6351 0.0.0.0:* LISTEN 13758/redis-server
tcp 0 0 192.168.8.51:16351 0.0.0.0:* LISTEN 13758/redis-server
2、51主机作集群管理主机,提供管理集群的ruby脚本(ruby-trib.rb)
[root@host51 ~]# ls
bin redis-4.0.8 redis-4.0.8.tar.gz redis-cluster
[root@host51 ~]# cd redis-cluster/
[root@host51 redis-cluster]# ls //准备如下软件包
redis-3.2.1.gem ruby-devel-2.0.0.648-30.el7.x86_64.rpm
[root@host51 redis-cluster]# yum -y install ruby
[root@host51 redis-cluster]# yum -y install ruby-devel-2.0.0.648-30.el7.x86_64.rpm
[root@host51 redis-cluster]# which gem
/usr/bin/gem
[root@host51 redis-cluster]# cd /root/redis-4.0.8/src
[root@host51 src]# ls *.rb
redis-trib.rb
[root@host51 src]# which $PATH //查看系统命令存放路径
/usr/bin/which: no bin in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root)
[root@host51 src]# mkdir /root/bin
[root@host51 src]# cp redis-trib.rb /root/bin
[root@host51 src]# chmod +x /root/bin/redis-trib.rb
[root@host51 src]# redis-trib.rb help
- redis-trib.rb脚本
语法格式:redis-trib 选项 参数
选项:
add-node 添加master主机
check 检测集群
reshard 重新分片
add-node --slave 添加是slave主机
del-node 删除主机
3、创建集群(redis集群固定三台主库)
[root@host51 ~]# redis-trib.rb create --replicas 1 192.168.8.51:6351 192.168.8.52:6352 192.168.8.53:6353 192.168.8.54:6354 192.168.8.55:6355 192.168.8.56:6356 //replicas选项为从库个数
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.8.51:6351
192.168.8.52:6352
192.168.8.53:6353
Adding replica 192.168.8.55:6355 to 192.168.8.51:6351 //51为主库,55为从库
Adding replica 192.168.8.56:6356 to 192.168.8.52:6352 //52为主库,56为从库
Adding replica 192.168.8.54:6354 to 192.168.8.53:6353 //53为主库,54为从库
M: 334ce3ed9703be74f417793a52b2232daeca44c1 192.168.8.51:6351
slots:0-5460 (5461 slots) master
M: 55e4362e028614e7433cf994db476277ca72b2ac 192.168.8.52:6352
slots:5461-10922 (5462 slots) master
M: 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e 192.168.8.53:6353
slots:10923-16383 (5461 slots) master
S: a657c77b6c720aa67315cb0599c97988c7a27d14 192.168.8.54:6354
replicates 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e
S: 963e5e46c7033e634c2e35852d2e4c4cca7e0e26 192.168.8.55:6355
replicates 334ce3ed9703be74f417793a52b2232daeca44c1
S: 2721a1c478642a4bb432bd4db4e17cee78faddfe 192.168.8.56:6356
replicates 55e4362e028614e7433cf994db476277ca72b2ac
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.8.51:6351)
M: 334ce3ed9703be74f417793a52b2232daeca44c1 192.168.8.51:6351
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: a657c77b6c720aa67315cb0599c97988c7a27d14 192.168.8.54:6354
slots: (0 slots) slave
replicates 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e
S: 963e5e46c7033e634c2e35852d2e4c4cca7e0e26 192.168.8.55:6355
slots: (0 slots) slave
replicates 334ce3ed9703be74f417793a52b2232daeca44c1
M: 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e 192.168.8.53:6353
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 2721a1c478642a4bb432bd4db4e17cee78faddfe 192.168.8.56:6356
slots: (0 slots) slave
replicates 55e4362e028614e7433cf994db476277ca72b2ac
M: 55e4362e028614e7433cf994db476277ca72b2ac 192.168.8.52:6352
slots:5461-10922 (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered. //表示集群创建成功
++++++++++++++++++++++++++++++++++
若集群创建失败,排错思路如下:
(1)检查各节点数据库是否为空(否则进入数据库执行flushall命令)
(2)检查各节点配置文件/etc/redis/6379.conf是否配置正确
(3)各节点执行rm -rf /var/lib/redis/6379/*,删除创建集群失败时生成的数据和节点信息
(4)各节点重启redis服务(redis-cli -h ip -p port shutdown,/etc/init.d/redis_6379 start)
(5)检查redis及集群监听端口是否开启(netstat -ntulp |grep port)
(6)管理主机上再重新执行创建集群命令
+++++++++++++++++++++++++++++++++
4、检查集群
[root@host51 ~]# redis-trib.rb check 192.168.8.56:6356 //检查集群节点角色(以任意节点查询)
>>> Performing Cluster Check (using node 192.168.8.56:6356)
S: 2721a1c478642a4bb432bd4db4e17cee78faddfe 192.168.8.56:6356
slots: (0 slots) slave
replicates 55e4362e028614e7433cf994db476277ca72b2ac
M: 55e4362e028614e7433cf994db476277ca72b2ac 192.168.8.52:6352
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e 192.168.8.53:6353
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 963e5e46c7033e634c2e35852d2e4c4cca7e0e26 192.168.8.55:6355
slots: (0 slots) slave
replicates 334ce3ed9703be74f417793a52b2232daeca44c1
S: a657c77b6c720aa67315cb0599c97988c7a27d14 192.168.8.54:6354
slots: (0 slots) slave
replicates 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e
M: 334ce3ed9703be74f417793a52b2232daeca44c1 192.168.8.51:6351
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@host51 ~]# redis-trib.rb info 192.168.8.56:6356 //查看集群节点信息
192.168.8.52:6352 (55e4362e...) -> 3 keys | 5462 slots | 1 slaves.
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 5461 slots | 1 slaves.
192.168.8.51:6351 (334ce3ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
5、客户端(50主机)访问集群
[root@host50 ~]# redis-cli -c -h 192.168.8.56 -p 6356
192.168.8.56:6356> keys *
1) "name"
2) "value"
192.168.8.56:6356> set dizhi hb
-> Redirected to slot [6842] located at 192.168.8.52:6352
OK
192.168.8.52:6352> exit
[root@host50 ~]# redis-cli -c -h 192.168.8.53 -p 6353
192.168.8.53:6353> keys *
(empty list or set)
192.168.8.53:6353> get dizhi
-> Redirected to slot [6842] located at 192.168.8.52:6352
"hb"
6、测试集群高可用
集群不能用的情况:
(1)有半数或者半数以上的master挂掉,集群就不能用了
(2)如果集群任意master挂掉,且当前master没有slave,集群不能用
注:一个主库挂掉,它的从库自动顶替为主库,正常使用(前提是:有半数或者半数以上的master能用),挂掉的主库修复好后,会成为从库,不会抢占为主
例:关闭52主机redis服务,则56主机升级为主库,且自动同步52主机数据库数据,当再次开启52主机redis服务时,52主机变为56主机的从库(演示略)
7、添加master主机(以57主机为例)
(1)57主机运行redis服务并启用集群模式,且数据库无数据
(2)在管理主机上作如下配置
[root@host51 ~]# redis-trib.rb add-node 192.168.8.57:6357 192.168.8.53:6353 //不指定角色时,默认新增主机为master,后一个参数为集群现有任意节点
[root@host51 ~]# redis-trib.rb check 192.168.8.53:6353
[root@host51 ~]# redis-trib.rb reshard 192.168.8.53:6353 //重新分片,分配hash槽
指定移出hash槽个数
指定接收hash槽主机ID
指定移出hash槽主机ID
移出策略:all
yes确认
[root@host51 ~]# redis-trib.rb check 192.168.8.53:6353 //检查添加57节点及分配hash槽后的节点角色
[root@host51 ~]# redis-trib.rb info 192.168.8.53:6353 //查看节点信息
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
192.168.8.51:6351 (334ce3ed...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.57:6357 (e88d4c38...) -> 1 keys | 4096 slots | 0 slaves. //57主机添加成功,有4096个hash槽,但没有从库
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
8、添加slave主机(以58主机为例)
(1)58主机运行redis服务并启用集群模式,且数据库无数据
(2)在管理主机上作如下配置
添加slave主机:
#redis-trib.rb add-node --slave [–master-id id值] ip地址:端口 任意已有主机ip:端口,如果不指定主节点id的话,默认将新节点添加为从节点最少的主节点的从节点
[root@host51 ~]# redis-trib.rb add-node --slave 192.168.8.58:6358 192.168.8.53:6353 //默认将58主机添加为从库最少的master主机(这里即57主机)
[root@host51 ~]# redis-trib.rb check 192.168.8.53:6353
>>> Performing Cluster Check (using node 192.168.8.53:6353)
M: 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e 192.168.8.53:6353
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 31ee5ff93141406f58e5ceb07654a1e2272e7fda 192.168.8.58:6358
slots: (0 slots) slave
replicates e88d4c38ceda706b55e354d818a5b20d7de290ad //58主机添加为57主机从库
M: 2721a1c478642a4bb432bd4db4e17cee78faddfe 192.168.8.56:6356
slots:6827-10922 (4096 slots) master
1 additional replica(s)
M: 334ce3ed9703be74f417793a52b2232daeca44c1 192.168.8.51:6351
slots:1365-5460 (4096 slots) master
1 additional replica(s)
M: e88d4c38ceda706b55e354d818a5b20d7de290ad 192.168.8.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
1 additional replica(s)
S: 55e4362e028614e7433cf994db476277ca72b2ac 192.168.8.52:6352
slots: (0 slots) slave
replicates 2721a1c478642a4bb432bd4db4e17cee78faddfe
S: 963e5e46c7033e634c2e35852d2e4c4cca7e0e26 192.168.8.55:6355
slots: (0 slots) slave
replicates 334ce3ed9703be74f417793a52b2232daeca44c1
S: a657c77b6c720aa67315cb0599c97988c7a27d14 192.168.8.54:6354
slots: (0 slots) slave
replicates 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@host51 ~]# redis-trib.rb info 192.168.8.53:6353 //57主机的从库58添加成功
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
192.168.8.51:6351 (334ce3ed...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.57:6357 (e88d4c38...) -> 1 keys | 4096 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
9、客户端测试连接redis数据库
[root@host50 ~]# redis-cli -c -h 192.168.8.53 -p 6353 //连接集群中的主机要添加-c选项
192.168.8.53:6353> set sex girl
-> Redirected to slot [2584] located at 192.168.8.51:6351
OK
192.168.8.51:6351> set age 25
-> Redirected to slot [741] located at 192.168.8.57:6357
OK
192.168.8.57:6357>
10、移除从节点(以移除58主机为例)
格式:
#redis-trib.rb del-node master-ip:port slave-id
slave-id值可以通过redis-trib.rb的check命令查出
[root@host51 ~]# redis-trib.rb del-node 192.168.8.51:6351 31ee5ff93141406f58e5ceb07654a1e2272e7fda
>>> Removing node 31ee5ff93141406f58e5ceb07654a1e2272e7fda from cluster 192.168.8.51:6351
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node. //移除节点同时,会关闭节点的redis服务
[root@host51 ~]# redis-trib.rb info 192.168.8.51:6351192.168.8.51:6351 (334ce3ed...) -> 1 keys | 4096 slots | 1 slaves.
192.168.8.57:6357 (e88d4c38...) -> 2 keys | 4096 slots | 0 slaves. //此时57的从节点为0
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
[OK] 5 keys in 4 masters.
0.00 keys per slot on average.
11、移除主节点
(1)释放hash槽,重新分片
[root@host51 ~]# redis-trib.rb reshard 192.168.8.51:6351
指定移出的hash槽个数
指定接收hash槽的主机ID
指定移出hash槽的主机ID
done
yes
(2)移除master节点
[root@host51 ~]# redis-trib.rb del-node 192.168.8.51:6351 e88d4c38ceda706b55e354d818a5b20d7de290ad
>>> Removing node e88d4c38ceda706b55e354d818a5b20d7de290ad from cluster 192.168.8.51:6351
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node. //移除节点同时,也会关闭节点redis服务
[root@host51 ~]# redis-trib.rb info 192.168.8.51:6351 //查看移除结果
192.168.8.51:6351 (334ce3ed...) -> 3 keys | 8192 slots | 1 slaves.
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
[OK] 5 keys in 3 masters.
0.00 keys per slot on average.
12、把之前移除的redis主机再添加到集群中(以57主机为例)
(1)启动57主机redis服务
[root@host57 ~]# /etc/init.d/redis_6379 start
Starting Redis server...
[root@host57 ~]# netstat -ntulp |grep 6357
tcp 0 0 192.168.8.57:6357 0.0.0.0:* LISTEN 14124/redis-server
tcp 0 0 192.168.8.57:16357 0.0.0.0:* LISTEN 14124/redis-server
(2)重置节点集群信息
[root@host57 ~]# redis-cli -h 192.168.8.57 -p 6357
192.168.8.57:6357> CLUSTER RESET
OK
192.168.8.57:6357> CLUSTER INFO
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:9
cluster_my_epoch:8
cluster_stats_messages_ping_sent:124
cluster_stats_messages_sent:124
cluster_stats_messages_pong_received:124
cluster_stats_messages_received:124
192.168.8.57:6357>
(3)在管理主机51上添加主节点(57)
[root@host51 ~]# redis-trib.rb add-node 192.168.8.57:6357 192.168.8.51:6351
>>> Adding node 192.168.8.57:6357 to cluster 192.168.8.51:6351
>>> Performing Cluster Check (using node 192.168.8.51:6351)
M: 334ce3ed9703be74f417793a52b2232daeca44c1 192.168.8.51:6351
slots:0-6826,10923-12287 (8192 slots) master
1 additional replica(s)
S: a657c77b6c720aa67315cb0599c97988c7a27d14 192.168.8.54:6354
slots: (0 slots) slave
replicates 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e
S: 963e5e46c7033e634c2e35852d2e4c4cca7e0e26 192.168.8.55:6355
slots: (0 slots) slave
replicates 334ce3ed9703be74f417793a52b2232daeca44c1
M: 6eb7b9ec6e43eca8a0665131d90130f8d1afe96e 192.168.8.53:6353
slots:12288-16383 (4096 slots) master
1 additional replica(s)
M: 2721a1c478642a4bb432bd4db4e17cee78faddfe 192.168.8.56:6356
slots:6827-10922 (4096 slots) master
1 additional replica(s)
S: 55e4362e028614e7433cf994db476277ca72b2ac 192.168.8.52:6352
slots: (0 slots) slave
replicates 2721a1c478642a4bb432bd4db4e17cee78faddfe
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.8.57:6357 to make it join the cluster.
[OK] New node added correctly.
[root@host51 ~]# redis-trib.rb info 192.168.8.51:6351
192.168.8.51:6351 (334ce3ed...) -> 3 keys | 8192 slots | 1 slaves.
192.168.8.57:6357 (e88d4c38...) -> 0 keys | 0 slots | 0 slaves.
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
[OK] 5 keys in 4 masters.
0.00 keys per slot on average.
(4)重新分片,分配hash槽
[root@host51 ~]# redis-trib.rb reshard 192.168.8.51:6351
指定移出的hash槽个数
指定接收hash槽的主机ID
指定移出hash槽的主机ID
done
yes
[root@host51 ~]# redis-trib.rb info 192.168.8.51:6351
192.168.8.51:6351 (334ce3ed...) -> 1 keys | 4096 slots | 1 slaves.
192.168.8.57:6357 (e88d4c38...) -> 2 keys | 4096 slots | 0 slaves.
192.168.8.53:6353 (6eb7b9ec...) -> 0 keys | 4096 slots | 1 slaves.
192.168.8.56:6356 (2721a1c4...) -> 2 keys | 4096 slots | 1 slaves.
[OK] 5 keys in 4 masters.
0.00 keys per slot on average.
13、把之前移除的redis主机再添加到集群中(以58主机添加为57的从节点为例)
(1)启动58主机redis服务
(2)重置节点集群信息
(3)添加从节点(参照之前添加从节点的命令)
14、当一组主从全部down掉,则整个集群无法工作(无法存储数据),因此需要保证一组主从中至少一台主机正常工作,通过恢复down掉的那组redis主机(先入为主),可以使集群恢复正常工作
15、将所有redis主机恢复为独立的数据库服务器
(1)将redis主机移除集群(注意主从移除的区别)
(2)修改配置文件/etc/redis/6379.conf(注释第815、823、829行),禁用集群功能
(3)清空redis数据库目录(/var/lib/redis/6379/)下的文件
(4)启动服务,查看端口(只有redis服务端口,没有集群端口)
(5)登录redis服务器(redis-cli -h ip -p port),测试存储数据