Docker安装Redis 6.2.6的3主3从集群和集群的扩缩容操作

1. 运行6个redis容器

运行6个redis容器，容器的name、port、data挂载宿主机目录不一样。name分别为redis1 ~ redis6，port分别为6381 ~ 6386，data挂载宿主机目录分别为/root/redis1/data ~ /root/redis6/data

这里只列出了redis1和redis2的，其它4个redis容器类似

[root@docker ~]# 
[root@docker ~]# docker run --net host --name redis1 --privileged=true \
> -v /root/redis1/data:/data \
> -d redis \
> redis-server \
> --bind 0.0.0.0  --port 6381 \
> --appendonly yes --cluster-enabled yes
031f6f270fb3181a47de0ca709c7d1962041c1bed93fb70a8d9e63bd9a21386f
[root@docker ~]# 
[root@docker ~]# docker run --net host --name redis2 --privileged=true \
> -v /root/redis2/data:/data \
> -d redis \
> redis-server \
> --bind 0.0.0.0  --port 6382 \
> --appendonly yes --cluster-enabled yes
2461dcd5ab1c7c853ba3b817d70f6c5675d5962e30f9d72bb2287f104b8cc31e
[root@docker ~]# 

参数说明：

• –net：默认方式，使用宿主机的IP和端口
• –bind：这里允许所有的客户端IP能连接到redis-server
• –cluster-enabled：是否开启redis集群
• –port：定义redis-server的端口号

2. redis cluster创建

进入任意一台redis容器，创建redis集群。–cluster-replicas等于1：表示集群的一个master会有一个slave, 最后形成一个3主3从的集群。会为每个master节点分配一定的slot

[root@docker ~]# 
[root@docker ~]# docker exec -it 031f6f270fb3 bash
root@docker:/data# 
root@docker:/data# redis-cli --cluster create 192.168.23.31:6381 192.168.23.31:6382 192.168.23.31:6383 192.168.23.31:6384 192.168.23.31:6385 192.168.23.31:6386 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.23.31:6385 to 192.168.23.31:6381
Adding replica 192.168.23.31:6386 to 192.168.23.31:6382
Adding replica 192.168.23.31:6384 to 192.168.23.31:6383
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[0-5460] (5461 slots) master
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[5461-10922] (5462 slots) master
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[10923-16383] (5461 slots) master
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@docker:/data#

如果master挂掉，master对应的slave会变成master。再启动挂掉的master，该master会变成slave

3. 查看集群信息

3.1 cluster info

root@docker:/data# redis-cli -h 192.168.23.31 -p 6381 -c
192.168.23.31:6381> 
192.168.23.31:6381> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:253
cluster_stats_messages_pong_sent:247
cluster_stats_messages_sent:500
cluster_stats_messages_ping_received:242
cluster_stats_messages_pong_received:253
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:500
192.168.23.31:6381> 

可以看到cluster_state:ok, cluster_slots_ok:16384, cluster_knows_nodes:6这些信息

3.2 cluster nodes

192.168.23.31:6381> 
192.168.23.31:6381> cluster nodes
3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384@16384 slave f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 0 1651234706564 2 connected
371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381@16381 myself,master - 0 1651234707000 1 connected 0-5460
a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385@16385 slave 7aa2741496275e87241ec85ca72e775caf22406a 0 1651234706000 3 connected
f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382@16382 master - 0 1651234704492 2 connected 5461-10922
7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383@16383 master - 0 1651234706000 3 connected 10923-16383
9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386@16386 slave 371ad1e8bfccf2059abed694cab0073837a27432 0 1651234707605 1 connected
192.168.23.31:6381> 

可以看到3个master和3个slave, 其中端口为6381这台也是myself

3.3 cluster check

root@docker:/data# 
root@docker:/data# redis-cli --cluster check 192.168.23.31:6381
192.168.23.31:6381 (371ad1e8...) -> 0 keys | 5461 slots | 1 slaves.
192.168.23.31:6382 (f60092a5...) -> 0 keys | 5462 slots | 1 slaves.
192.168.23.31:6383 (7aa27414...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@docker:/data# 

可以看到3台master和3台slave的IP和端口，和3台master分配的slot的范围

4. 集群的扩容

再运行2个redis容器。name分别为redis7、redis8，port分别为6387、6388，data挂载宿主机目录分别为/root/redis7/data、/root/redis8/data

[root@docker ~]# 
[root@docker ~]# docker run --net host --name redis7 --privileged=true \
> -v /root/redis7/data:/data \
> -d redis \
> redis-server \
> --bind 0.0.0.0  --port 6387 \
> --appendonly yes --cluster-enabled yes
961e32fd04cf4b66e0910f59d9aa66c8196a9341aa5a5116e490d7d5006fb098
[root@docker ~]# 
[root@docker ~]# docker run --net host --name redis8 --privileged=true \
> -v /root/redis8/data:/data \
> -d redis \
> redis-server \
> --bind 0.0.0.0  --port 6388 \
> --appendonly yes --cluster-enabled yes
db4e7c8c341be44aa46fc7921c7e79432acb588d968e33bb27a43db23b64f72b
[root@docker ~]# 

通过集群中端口为6381的master，将端口为6387的redis添加到集群中作为master。此时端口为6387的master还未分配slot

root@docker:/data# 
root@docker:/data# redis-cli --cluster add-node 192.168.23.31:6387 192.168.23.31:6381
>>> Adding node 192.168.23.31:6387 to cluster 192.168.23.31:6381
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.23.31:6387 to make it join the cluster.
[OK] New node added correctly.
root@docker:/data# 

为端口为6387的redis分配slot。移动4096个slot到端口为6387的redis，选择all表示从其它所有的master中均匀的分配一些slot到目标redis master, 因为全部重新分配slot的成本太高

root@docker:/data# 
root@docker:/data# redis-cli --cluster reshard 192.168.23.31:6381
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[10923-16383] (5462 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
M: f3d2cf352c34c91b44142a00d5fa0e925226f7b5 192.168.23.31:6387
   slots: (0 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? f3d2cf352c34c91b44142a00d5fa0e925226f7b5
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all
......省略部分......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......省略部分......

为端口6387的redis master分配一个端口6388的slave。其中–cluster-master-id指定端口6387的master的ID

root@docker:/data# 
root@docker:/data# redis-cli --cluster add-node 192.168.23.31:6388 192.168.23.31:6387 --cluster-slave --cluster-master-id f3d2cf352c34c91b44142a00d5fa0e925226f7b5 
>>> Adding node 192.168.23.31:6388 to cluster 192.168.23.31:6387
>>> Performing Cluster Check (using node 192.168.23.31:6387)
M: f3d2cf352c34c91b44142a00d5fa0e925226f7b5 192.168.23.31:6387
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.23.31:6388 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 192.168.23.31:6387.
[OK] New node added correctly.
root@docker:/data# 
root@docker:/data# redis-cli --cluster check 192.168.23.31:6381
192.168.23.31:6381 (371ad1e8...) -> 0 keys | 4096 slots | 1 slaves.
192.168.23.31:6382 (f60092a5...) -> 0 keys | 4096 slots | 1 slaves.
192.168.23.31:6383 (7aa27414...) -> 0 keys | 4096 slots | 1 slaves.
192.168.23.31:6387 (f3d2cf35...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 0 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: f60ac30bb25570abc0c9bc85c25e0e9b92798646 192.168.23.31:6388
   slots: (0 slots) slave
   replicates f3d2cf352c34c91b44142a00d5fa0e925226f7b5
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
M: f3d2cf352c34c91b44142a00d5fa0e925226f7b5 192.168.23.31:6387
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@docker:/data# 

5. 集群的缩容

目标是从集群中删除端口为6387和6388的这两个redis

先删除端口为6388的redis slave。其中指定的节点ID是端口为6388的redis节点ID

root@docker:/data# 
root@docker:/data# redis-cli --cluster del-node 192.168.23.31:6388 f60ac30bb25570abc0c9bc85c25e0e9b92798646
>>> Removing node f60ac30bb25570abc0c9bc85c25e0e9b92798646 from cluster 192.168.23.31:6388
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
root@docker:/data# 

将端口为6387的redis master的4096个slot，分配给端口为6381的redis master。其中done表示输入slot源编写完成

root@docker:/data# redis-cli --cluster reshard 192.168.23.31:6381
>>> Performing Cluster Check (using node 192.168.23.31:6381)
M: 371ad1e8bfccf2059abed694cab0073837a27432 192.168.23.31:6381
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 3335d1ed096121aa7a87c7a61285f08cbb935425 192.168.23.31:6384
   slots: (0 slots) slave
   replicates f60092a586a0a407ff7594124ec0f4ae7bb9a7a8
S: a1d333045f97a093bf674dd0b0014dc45751bedc 192.168.23.31:6385
   slots: (0 slots) slave
   replicates 7aa2741496275e87241ec85ca72e775caf22406a
M: f60092a586a0a407ff7594124ec0f4ae7bb9a7a8 192.168.23.31:6382
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
M: 7aa2741496275e87241ec85ca72e775caf22406a 192.168.23.31:6383
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 9bbeed41e6ad84ec9bb1e5cc8682d06b1a397f62 192.168.23.31:6386
   slots: (0 slots) slave
   replicates 371ad1e8bfccf2059abed694cab0073837a27432
M: f3d2cf352c34c91b44142a00d5fa0e925226f7b5 192.168.23.31:6387
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 371ad1e8bfccf2059abed694cab0073837a27432
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: f3d2cf352c34c91b44142a00d5fa0e925226f7b5
Source node #2: done
......省略部分......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......省略部分......

再删除端口为6387的redis master。其中指定的节点ID是端口为6387的redis节点ID

root@docker:/data# 
root@docker:/data# redis-cli --cluster del-node 192.168.23.31:6387 f3d2cf352c34c91b44142a00d5fa0e925226f7b5 
>>> Removing node f3d2cf352c34c91b44142a00d5fa0e925226f7b5 from cluster 192.168.23.31:6387
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
root@docker:/data#