Redis Cluster 集群维护

一、动态添加Node节点

注意事项：

1. 需要与之前的Redis node版本相同
2. 需要与之前的Redis node配置一致
3. 分别启动两台Redis node，一主一从

1.1 修改新redis的配置文件

root@ubuntu1804:/usr/local/src# vim /apps/redis/etc/redis.conf
masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf
root@ubuntu1804:/usr/local/src# systemctl restart redis.service

1.2 添加master节点到集群

1.2.1 redis 4

redis-trib.rb add-node 10.0.0.37:6379 10.0.0.31:6379

1.2.2 redis 5

redis-cli -a 123456 --cluster add-node 192.168.7.104:6379 192.168.7.101:6379

1.3 重新分配槽位

1.3.1 redis 4

root@ubuntu1804:~# redis-trib.rb check 10.0.0.31:6379    #查看当前槽位分配情况

#reshard划分槽位：从某一个node减少槽位添加到新的node
#rebalance划分槽位：全部重新平均分配
#使用reshard前提是集群里面没有数据，先把数据导出，再删除

root@ubuntu1804:~# redis-trib.rb reshard 10.0.0.37:6379
>>> Performing Cluster Check (using node 10.0.0.37:6379)
M: 54faa689bf5c6057aa3c810a988b90c08b0c06d2 10.0.0.37:6379
   slots: (0 slots) master
   0 additional replica(s)
M: 159eadcb486b6b2b5956281ddf47d9d75b0a22be 10.0.0.33:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 11236eab81bdde236455a16494537a6b83e0fc78 10.0.0.35:6379
   slots: (0 slots) slave
   replicates 0d3ba938ff620d3ddf909698f383757e4e311ebf
S: f44add80c6b2179d93eb68c4169238da94d66476 10.0.0.34:6379
   slots: (0 slots) slave
   replicates 159eadcb486b6b2b5956281ddf47d9d75b0a22be
M: 0d3ba938ff620d3ddf909698f383757e4e311ebf 10.0.0.31:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5 10.0.0.32:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 5d48beb1ce78346849178cf060e3b08a7ddd7100 10.0.0.36:6379
   slots: (0 slots) slave
   replicates 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 54faa689bf5c6057aa3c810a988b90c08b0c06d2
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:all
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......

root@ubuntu1804:~# redis-trib.rb check 10.0.0.31:6379    #重新check，查看10.0.0.37的槽位是否发生改变
>>> Performing Cluster Check (using node 10.0.0.31:6379)
M: 0d3ba938ff620d3ddf909698f383757e4e311ebf 10.0.0.31:6379
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: 54faa689bf5c6057aa3c810a988b90c08b0c06d2 10.0.0.37:6379
   slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   0 additional replica(s)
S: f44add80c6b2179d93eb68c4169238da94d66476 10.0.0.34:6379
   slots: (0 slots) slave
   replicates 159eadcb486b6b2b5956281ddf47d9d75b0a22be
M: 159eadcb486b6b2b5956281ddf47d9d75b0a22be 10.0.0.33:6379
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
M: 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5 10.0.0.32:6379
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)
S: 11236eab81bdde236455a16494537a6b83e0fc78 10.0.0.35:6379
   slots: (0 slots) slave
   replicates 0d3ba938ff620d3ddf909698f383757e4e311ebf
S: 5d48beb1ce78346849178cf060e3b08a7ddd7100 10.0.0.36:6379
   slots: (0 slots) slave
   replicates 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

#结束节后，再导入数据

1.3.2 redis 5

redis-cli -a 123456 --cluster reshard 192.168.7.104:6379

1.4 为新的master添加slave节点

1.4.1 方式一：命令方式添加slave节点并指定master-id

root@ubuntu1804:~# redis-trib.rb add-node --slave --master-id 54faa689bf5c6057aa3c810a988b90c08b0c06d2 10.0.0.38:6379 10.0.0.31:6379

1.4.2 方式二：先添加节点，后续再改为slave

redis-trib.rb add-node 10.0.0.38:6379 10.0.0.31:6379
redis-cli -h 10.0.0.38 -p 6379 -a 123456    #登录到新添加节点
10.0.0.38:6379> CLUSTER NODES    #查看当前集群节点，找到目标master 的ID
10.0.0.38:6379> CLUSTER REPLICATE 886338acd50c3015be68a760502b239f4509881c    #将其设置 slave，命令格式为cluster replicate MASTERID
10.0.0.38:6379> CLUSTER NODES    #再次查看集群节点状态，验证节点是否已经更改为指定master 的slave

二、动态删除Node节点

添加节点的时候是先添加node节点到集群，然后分配槽位，删除节点的操作与添加节点的操作正好相反，是先将被删除的Redis node上的槽位迁移到集群中的其他Redis node节点上，然后再将其删除，如果一个Redis node节点上的槽位没有被完全迁移，删除该node的时候会提示有数据且无法删除。

2.1 修改新redis的配置文件，以master身份加入集群

root@ubuntu1804:/usr/local/src# vim /apps/redis/etc/redis.conf
masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf
root@ubuntu1804:/usr/local/src# systemctl restart redis.service

把master 10.0.0.37的所有槽位分给10.0.0.40

root@ubuntu1804:~# redis-trib.rb add-node 10.0.0.40:6379 10.0.0.31:6379
root@ubuntu1804:~# redis-trib.rb reshard 10.0.0.40:6379
root@ubuntu1804:~# redis-trib.rb reshard 10.0.0.40:6379
>>> Performing Cluster Check (using node 10.0.0.40:6379)
M: 8eea215e74e0ac5475ae97af8d5f3ec95a6bcae4 10.0.0.40:6379
   slots: (0 slots) master
   0 additional replica(s)
S: 11236eab81bdde236455a16494537a6b83e0fc78 10.0.0.35:6379
   slots: (0 slots) slave
   replicates 0d3ba938ff620d3ddf909698f383757e4e311ebf
M: 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5 10.0.0.32:6379
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)
S: 0eff14918dc43bf5a7e3e6ff227497802fd0ccf7 10.0.0.38:6379
   slots: (0 slots) slave
   replicates 8eea215e74e0ac5475ae97af8d5f3ec95a6bcae4
S: f44add80c6b2179d93eb68c4169238da94d66476 10.0.0.34:6379
   slots: (0 slots) slave
   replicates 159eadcb486b6b2b5956281ddf47d9d75b0a22be
S: 5d48beb1ce78346849178cf060e3b08a7ddd7100 10.0.0.36:6379
   slots: (0 slots) slave
   replicates 3eef4279bf52a4b4085cb87de75b8e3dd173e4c5
M: 159eadcb486b6b2b5956281ddf47d9d75b0a22be 10.0.0.33:6379
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
M: 54faa689bf5c6057aa3c810a988b90c08b0c06d2 10.0.0.37:6379
    slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   1 additional replica(s)
M: 0d3ba938ff620d3ddf909698f383757e4e311ebf 10.0.0.31:6379
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? 8eea215e74e0ac5475ae97af8d5f3ec95a6bcae4
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:54faa689bf5c6057aa3c810a988b90c08b0c06d2
Source node #2:done
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
......

2.2 移除node节点

root@ubuntu1804:~# redis-trib.rb del-node 10.0.0.31:6379 54faa689bf5c6057aa3c810a988b90c08b0c06d2
>>> Removing node 54faa689bf5c6057aa3c810a988b90c08b0c06d2 from cluster 10.0.0.31:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

2.3 查看master与slave的对应关系

其中原来master 10.0.0.37 已经变成 10.0.0.40

root@ubuntu1804:~# redis-trib.rb info 10.0.0.31:6379
10.0.0.31:6379 (0d3ba938...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.33:6379 (159eadcb...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.32:6379 (3eef4279...) -> 0 keys | 4096 slots | 1 slaves.
10.0.0.40:6379 (8eea215e...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 0 keys in 4 masters.
0.00 keys per slot on average.

三、导入Redis数据

导入数据之前需要关闭各redis 服务器的密码，包括集群中的各node和源Redis server，避免认证带来的环境不一致从而无法导入，可以加参数–cluster-replace 强制替换Redis cluster已有的key。

3.1 redis 3/4

redis-trib.rb import --from 10.0.0.31:6382 --replace 10.0.0.40:6379

3.2 redis 5

redis-cli --cluster import 10.0.0.31:6379 --cluster-from 10.0.0.40:6379 --cluster-copy

四、Redis 集群扩展解决方案

4.1 codis

Codis 是一个分布式 Redis 解决方案, 对于上层的应用来说, 连接到 Codis Proxy 和连接原生的 Redis Server 没有显著区别 (令不支持的命列表), 上层应用可以像使用单机的 Redis 一样使用, Codis 底层会处理请求的转发, 不停机的数据迁移等工作, 所有后边的一切事情, 对于前面的客户端来说是透明的, 可以简单的认为后边连接的是一个内存无限大的 Redis 服务。
codis-proxy相当于redis，即连接codis-proxy和连接redis是没有任何区别的，codis-proxy无状态，不负责记录是否在哪保存，数据在zookeeper记录，即codis proxy向zookeeper查询key的记录位置，proxy 将请求转发到一个组进行处理，一个组里面有一个master和一个或者多个slave组成，默认有1024个槽位，redis cluster 默认有16384个槽位，其把不同的槽位的内容放在不同的group。

4.2 twemproxy

由Twemproxy双向代理客户端实现分片，即代替用户将数据分片并到不同的后端服务器进行读写，其还支持memcached，可以为proxy配置算法，缺点为twemproxy是瓶颈，不支持数据迁移。