生产级Redis 哨兵 Cluster部署

原创已于 2023-07-04 09:17:02 修改 · 546 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#redis #数据库 #缓存

于 2023-07-03 22:18:27 首次发布

文章详细介绍了如何在Linux环境中部署RedisCluster，包括关闭防火墙、安装依赖、配置Redis、创建和验证集群以及处理故障转移。重点讨论了数据分片、高可用性和横向扩展的概念，并展示了通过redis-trib.rb工具进行集群管理的步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

简述：

Redis Cluster是Redis的分布式解决方案，用于在多个节点上分布和复制数据，提供高可用性和横向扩展能力。

下面是生产级Redis Cluster的详细介绍： 1. 数据分片：Redis Cluster将数据分成多个槽位（slot），默认有16384个槽位。每个节点负责管理一部分槽位，确保数据在集群中均匀分布。

2. 数据复制：每个槽位在集群中有一个主节点和多个从节点。主节点负责处理读写请求，从节点复制主节点的数据，并在主节点故障时接管主节点的角色。

3. 故障转移：当主节点故障时，Redis Cluster会自动进行故障转移，将一个从节点提升为主节点，并重新分配槽位。这个过程是自动化的，无需人工干预。

4. 高可用性：Redis Cluster通过数据复制和故障转移提供高可用性。即使有节点故障，集群仍然可以继续提供服务，不会导致数据丢失或服务中断。

5. 横向扩展：通过增加节点，可以实现Redis Cluster的横向扩展。新节点加入集群后，会自动接管一部分槽位，从而实现数据的平衡和负载均衡。

6. 客户端路由：客户端与Redis Cluster交互时，需要进行槽位的路由。客户端根据数据的key计算槽位，并将请求发送到对应的节点。这样可以保证相同的key总是被路由到同一个节点。

7. 集群管理：Redis Cluster提供了一些命令和工具，用于管理集群的状态和配置。可以查看节点信息、槽位分配情况，添加或删除节点，进行集群扩容或缩容等操作。

生产级Redis Cluster是一个可靠、高可用、可扩展的分布式存储解决方案，适用于处理大规模数据和高并发访问的场景。它通过数据分片、数据复制、故障转移等机制，保证数据的可靠性和服务的连续性。同时，它还提供了方便的管理和监控工具，简化了集群的部署和运维。

环境准备

关闭防火墙规则    iptables -F    setenforce 0    systemctl stop firewalld

规划主机名

master       192.168.1.129

slave          192.168.1.139

开始

安装依赖

[root@master ~]# yum -y install wget gcc gcc-c++ make tar openssl openssl-devel cmake

解包（下载好的）

[root@master ~]# tar xf redis-4.0.10.tar.gz -C /usr/src/

[root@master ~]# cd /usr/src/redis-4.0.10/

解包

[root@master redis-4.0.10]# make

[root@master redis-4.0.10]# make MALLOC=jemalloc    避免碎片和可伸缩并发支持

[root@master redis-4.0.10]# make PREFIX=/usr/local/redis install

将编译好的Redis安装到指定的目录`/usr/local/redis`下。

[root@master redis-4.0.10]# mkdir -p /usr/local/redis/conf

创建一个目录`/usr/local/redis/conf`，用于存放Redis的配置文件。

[root@master redis-4.0.10]# cp sentinel.conf /usr/local/redis/conf/
[root@master redis-4.0.10]# cp src/redis-trib.rb /usr/local/redis/bin/

将Redis哨兵配置文件（sentinel.conf）和Redis集群工具（redis-trib.rb）复制到指定的目录下。其中，sentinel.conf是用于配置Redis哨兵的文件，redis-trib.rb是用于管理Redis集群的工具。

[root@master redis-4.0.10]# ln -s /usr/local/redis/bin/* /usr/local/bin/

将/usr/local/redis/bin/目录下的所有文件都创建软链接到/usr/local/bin/目录下，以便在任何位置都可以直接执行这些文件。

配置

配置文件精简

[root@master redis-4.0.10]# cp redis.conf /usr/local/redis/conf/sentinel.conf

[root@master redis-4.0.10]# cd /usr/local/redis/

[root@master conf]# cp redis.conf{,.bak}

[root@master conf]# egrep -v "^$|^#" redis.conf.bak > redis.conf

1. 第一行命令将redis.conf文件复制到/usr/local/redis/conf/目录下。

2. 第二行命令进入/usr/local/redis/目录。

3. 第三行命令将redis.conf.bak文件复制为redis.conf.bak，即创建一个备份文件。

4. 第四行命令使用egrep命令过滤掉空行和以"#"开头的注释行，并将结果输出到redis.conf文件中，即将备份文件中的注释行去除，生成一个新的配置文件。

系统调优配置

[root@master conf]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

禁用透明大页功能，透明大页是一种内存管理技术，禁用可以提高系统性能。
[root@master conf]# echo never > /sys/kernel/mm/transparent_hugepage/defrag

禁用透明大页的碎片整理功能，同样可以提高系统性能。
[root@master conf]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

将禁用透明大页的命令添加到系统启动脚本中，确保系统每次启动时都会执行该命令。
[root@master conf]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local

将禁用透明大页碎片整理的命令添加到系统启动脚本中。
[root@master conf]# echo "* - nofile 10240" >> /etc/security/limits.conf

将最大打开文件数限制设置为10240，这可以提高系统对并发连接的支持能力。
[root@master conf]# echo "net.core.somaxconn = 10240" >> /etc/sysctl.conf

将TCP连接的最大排队数设置为10240，同样可以提高系统对并发连接的支持能力。
[root@master conf]# echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf

将内存过量分配策略设置为1，这可以避免内存不足时导致系统崩溃。

这些操作旨在优化系统性能和提高系统对并发连接的支持能力

[root@master conf]# sysctl -p
net.core.somaxconn = 10240
vm.overcommit_memory = 1

执行"sysctl -p"命令，可以将这些参数的设置应用到系统中

1. net.core.somaxconn = 10240：这是指定系统中每个监听套接字的最大连接数。这里设置为10240，表示最大连接数为10240个。

2. vm.overcommit_memory = 1：这是指定系统内存分配策略的参数。这里设置为1，表示系统会允许超额分配内存，即当系统内存不足时，仍然允许分配更多的内存。

多实例配置

[root@master conf]# mkdir -p /data/redis-cluster

创建一个名为`redis-cluster`的文件夹，并将其路径设置为`/data/redis-cluster`。如果该文件夹已存在，则不会报错。
[root@master conf]# cd /data/redis-cluster

切换当前工作目录到`/data/redis-cluster`。
[root@master redis-cluster]# mkdir 7020 7030 7040

在当前目录下创建三个文件夹，分别命名为`7020`、`7030`和`7040`。
[root@master redis-cluster]# cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7020/
[root@master redis-cluster]# cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7030/
[root@master redis-cluster]# cp /usr/local/redis/conf/redis.conf /data/redis-cluster/7040/

将`sentinel.conf`文件复制到每个子目录中。

安装 tree yum -y install tree 树状查看

[root@master 7040]# tree /data
/data
└── redis-cluster
    ├── 7020
    │   └── redis.conf
    ├── 7030
    │   └── redis.conf
    └── 7040
        └── redis.conf

4 directories, 3 files

开始验证

在lsave主机上操作

[root@slave ~]# mkdir -p /data/redis-cluster
[root@slave ~]# cd /data/redis-cluster
[root@slave redis-cluster]# mkdir 8022 8033 8044

再去master上把文件复制过来

[root@master redis-cluster]# scp 7020/redis.conf 192.168.1.129:/data/redis-cluster/8022/
root@192.168.1.129's password:
redis.conf                                                           100% 1313     1.4MB/s   00:00
[root@master redis-cluster]# scp 7020/redis.conf 192.168.1.129:/data/redis-cluster/8033/
root@192.168.1.129's password:
redis.conf                                                           100% 1313     1.8MB/s   00:00
[root@master redis-cluster]# scp 7020/redis.conf 192.168.1.129:/data/redis-cluster/8044/
root@192.168.1.129's password:
redis.conf                                                           100% 1313     1.7MB/s   00:00

查看

[root@slave 8022]# tree /data/
/data/
└── redis-cluster
    ├── 8022
    │   └── redis.conf
    ├── 8033
    │   └── redis.conf
    └── 8044
        └── redis.conf

4 directories, 3 files

修改master和slave的redis.conf配置文件

配置参考模板，修改配置文件的对应端口

[root@master redis-cluster]# vim 7020/redis.conf 

cluster-enabled yes
bind 0.0.0.0
port 7020
pidfile /data/redis-cluster/7020/redis.pid
logfile "/data/redis-cluster/7020/redis.log"
dir /data/redis-cluster/7020/
tcp-backlog 1024
timeout 0
tcp-keepalive 0
daemonize yes
loglevel notice
databases 16
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

启动

[root@master redis-cluster]# redis-server /data/redis-cluster/7020/redis.conf

[root@master redis-cluster]# redis-server /data/redis-cluster/7030/redis.conf
[root@master redis-cluster]# redis-server /data/redis-cluster/7040/redis.conf

查看
[root@master redis-cluster]# netstat -antup | grep redis
tcp        0      0 0.0.0.0:17020           0.0.0.0:*               LISTEN      66782/redis-server
tcp        0      0 0.0.0.0:7040            0.0.0.0:*               LISTEN      66796/redis-server
tcp        0      0 0.0.0.0:17030           0.0.0.0:*               LISTEN      66791/redis-server
tcp        0      0 0.0.0.0:7020            0.0.0.0:*               LISTEN      66782/redis-server
tcp        0      0 0.0.0.0:17040           0.0.0.0:*               LISTEN      66796/redis-server
tcp        0      0 0.0.0.0:7030            0.0.0.0:*               LISTEN      66791/redis-server

编译安装高ruby版本，Ruby版本需要大于等于2.2.2（yum安装的不符合）

tar xf ruby-2.5.1.tar.gz -C /usr/src/

cd /usr/src/ruby-2.5.1/

[root@master ruby-2.5.1]# ./configure && make && make install

[root@master ruby-2.5.1]# ruby --version
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]

在线安装ruby的redis扩展

[root@master ~]# /usr/local/bin/gem install -l redis-3.2.0.gem
Successfully installed redis-3.2.0
Parsing documentation for redis-3.2.0
Installing ri documentation for redis-3.2.0
Done installing documentation for redis after 0 seconds
1 gem installed

创建集群

[root@master ~]# redis-trib.rb create 127.0.0.1:7020 127.0.0.1:7030 127.0.0.1:7040
>>> Creating cluster
/usr/local/lib/ruby/gems/2.5.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
127.0.0.1:7020
127.0.0.1:7030
127.0.0.1:7040
M: 854d1a44759ebdbe6e5e5be9e6be05c52b0186f2 127.0.0.1:7020
   slots:0-5460 (5461 slots) master
M: 52af837845c847eff425349c02ff9177bec4cc36 127.0.0.1:7030
   slots:5461-10922 (5462 slots) master
M: 7b45331663855d2cf0b89eee1e63a9822f676213 127.0.0.1:7040
   slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.
>>> Performing Cluster Check (using node 127.0.0.1:7020)
M: 854d1a44759ebdbe6e5e5be9e6be05c52b0186f2 127.0.0.1:7020
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: 52af837845c847eff425349c02ff9177bec4cc36 127.0.0.1:7030
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: 7b45331663855d2cf0b89eee1e63a9822f676213 127.0.0.1:7040
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

这是一个用于创建Redis集群的命令。其中，"redis-trib.rb"是Redis集群管理工具的名称，"create"是指定创建集群的操作，而后面的"127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002"是指定了要创建集群的节点地址和端口号。具体来说，这个命令表示要在本地主机上创建一个包含三个节点的Redis集群，分别对应的地址和端口号是127.0.0.1:7000、127.0.0.1:7001和127.0.0.1:7002。

查看节点状态

[root@master ~]# redis-cli -p 7020 cluster nodes
52af837845c847eff425349c02ff9177bec4cc36 127.0.0.1:7030@17030 master - 0 1688389712441 2 connected 5461-10922
7b45331663855d2cf0b89eee1e63a9822f676213 127.0.0.1:7040@17040 master - 0 1688389711437 3 connected 10923-16383
854d1a44759ebdbe6e5e5be9e6be05c52b0186f2 127.0.0.1:7020@17020 myself,master - 0 1688389711000 1 connected 0-5460

[root@master ~]# ls /data/redis-cluster/7020/
nodes.conf redis.conf redis.log redis.pid

[root@master ~]# cat /data/redis-cluster/7020/nodes.conf
52af837845c847eff425349c02ff9177bec4cc36 127.0.0.1:7030@17030 master - 0 1688389645134 2 connected 5461-10922
7b45331663855d2cf0b89eee1e63a9822f676213 127.0.0.1:7040@17040 master - 0 1688389644934 3 connected 10923-16383
854d1a44759ebdbe6e5e5be9e6be05c52b0186f2 127.0.0.1:7020@17020 myself,master - 0 0 1 connected 0-5460
vars currentEpoch 3 lastVoteEpoch 0

使用redis-cli去操作集群，需要加入-c参数

redis-cli -c: Enable cluster mode (follow -ASK and -MOVED redirections).

[root@master ~]# redis-cli -c -p 7020
127.0.0.1:7020> set name crushlinux
-> Redirected to slot [5798] located at 127.0.0.1:7030
OK
127.0.0.1:7030> exit

去中心化，随意一个入口
[root@master ~]# redis-cli -c -p 7020 get name
"crushlinux"
[root@master ~]# redis-cli -c -p 7030 get name
"crushlinux"
[root@master ~]# redis-cli -c -p 7040 get name
"crushlinux"

slave上操作启动

[root@slave redis-4.0.10]# redis-server /data/redis-cluster/8022/redis.conf
[root@slave redis-4.0.10]# redis-server /data/redis-cluster/8033/redis.conf
[root@slave redis-4.0.10]# redis-server /data/redis-cluster/8044/redis.conf

查看只有7030成功看到其他都被拒绝

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7020 get name
Could not connect to Redis at 127.0.0.1:7030: Connection refused
Could not connect to Redis at 127.0.0.1:7030: Connection refused
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7030 get name
"crushlinux"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7040 get name
Could not connect to Redis at 127.0.0.1:7030: Connection refused
Could not connect to Redis at 127.0.0.1:7030: Connection refused

以远程的方式访问redis-cluster时，被拒绝是因为我们创建集群的时候，是以127.0.0.1的本地IP方式创建的，因此只有本地访问时cluster集群才能发挥作用。

远程访问7001端口时，之所以能成功，是因为数据本来就是存在了监听7001端口的redis上，所以我们不加-c也能访问成功。

cluster生产集群部署

在master上操作

列出7020端口的Redis集群节点目录下的文件。

[root@master ~]# ls /data/redis-cluster/7020
nodes.conf redis.conf redis.log redis.pid
[root@master ~]# ls /data/redis-cluster/7030
nodes.conf redis.conf redis.log redis.pid
[root@master ~]# ls /data/redis-cluster/7040
nodes.conf redis.conf redis.log redis.pid

删除7020端口的Redis集群节点目录下的nodes.conf文件。
[root@master ~]# rm -rf /data/redis-cluster/7020/nodes.conf
[root@master ~]# rm -rf /data/redis-cluster/7030/nodes.conf
[root@master ~]# rm -rf /data/redis-cluster/7040/nodes.conf

关闭7020端口的Redis进程。
[root@master ~]# redis-cli -p 7020 shutdown
[root@master ~]# redis-cli -p 7030 shutdown
[root@master ~]# redis-cli -p 7040 shutdown

启动7020端口的Redis进程，并使用指定的配置文件。
[root@master ~]# redis-server /data/redis-cluster/7020/redis.conf
[root@master ~]# redis-server /data/redis-cluster/7030/redis.conf
[root@master ~]# redis-server /data/redis-cluster/7040/redis.conf

重新创建redis-cluster集群

[root@master ~]# redis-trib.rb create 192.168.1.139:7020 192.168.1.139:7030 192.168.1.139:7040
>>> Creating cluster
/usr/local/lib/ruby/gems/2.5.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
192.168.1.139:7020
192.168.1.139:7030
192.168.1.139:7040
M: 45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020
   slots:0-5460 (5461 slots) master
M: 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030
   slots:5461-10922 (5462 slots) master
M: f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040
   slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.
>>> Performing Cluster Check (using node 192.168.1.139:7020)
M: 45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@master ~]# ps -ef | grep cluster | grep -v grep
root      81206      1  0 21:27 ?        00:00:00 redis-server 0.0.0.0:7020 [cluster]
root      81211      1  0 21:27 ?        00:00:00 redis-server 0.0.0.0:7030 [cluster]
root      81216      1  0 21:27 ?        00:00:00 redis-server 0.0.0.0:7040 [cluster]

在slave上进行远程连接cluster集群测试

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7020 set name crushlinux
OK
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7020 get name
"crushlinux"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7030 get name
"crushlinux"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7040 get name
"crushlinux"

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7040
192.168.1.139:7040> set aaa www 40创建后跳转到30了
-> Redirected to slot [10439] located at 192.168.1.139:7030
OK
192.168.1.139:7030> get aaa
"www"

批量导入数据观察key的集群分布情况

[root@slave redis-4.0.10]# for line in `seq -w 10000`;do redis-cli -h 192.168.1.139 -p 7020 -c set key3_${line} value_${line};done

分析cluster集群key的节点分布情况

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -p 7020 info Keyspace
# Keyspace
db0:keys=3337,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -p 7030 info Keyspace
# Keyspace
db0:keys=3355,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -p 7040 info Keyspace
# Keyspace
db0:keys=3310,expires=0,avg_ttl=0

并不是放在一个地方而是尽量平均

Cluster集群故障切换

redis-cluster集群从库的添加，master上操作

添加第三组从服务器

[root@master ~]# redis-trib.rb add-node --slave 192.168.1.129:8022 192.168.1.139:7020
>>> Adding node 192.168.1.129:8022 to cluster 192.168.1.139:7020
/usr/local/lib/ruby/gems/2.5.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing Cluster Check (using node 192.168.1.139:7020)
M: 45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.1.139:7020
>>> Send CLUSTER MEET to node 192.168.1.129:8022 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.1.139:7020.
[OK] New node added correctly.
[root@master ~]# redis-trib.rb add-node --slave 192.168.1.129:8033 192.168.1.139:7030
>>> Adding node 192.168.1.129:8033 to cluster 192.168.1.139:7030
/usr/local/lib/ruby/gems/2.5.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing Cluster Check (using node 192.168.1.139:7030)
M: 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
S: 550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022
   slots: (0 slots) slave
   replicates 45c468aa04a938bd65aa8c2c62a75321c08b4d25
M: 45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.1.139:7030
>>> Send CLUSTER MEET to node 192.168.1.129:8033 to make it join the cluster.
Waiting for the cluster to join.^[[A
>>> Configure node as replica of 192.168.1.139:7030.
[OK] New node added correctly.
[root@master ~]# redis-trib.rb add-node --slave 192.168.1.129:8044 192.168.1.139:7040
>>> Adding node 192.168.1.129:8044 to cluster 192.168.1.139:7040
/usr/local/lib/ruby/gems/2.5.0/gems/redis-3.2.0/lib/redis/client.rb:422: warning: constant ::Fixnum is deprecated
>>> Performing Cluster Check (using node 192.168.1.139:7040)
M: f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
M: 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
M: 45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022
   slots: (0 slots) slave
   replicates 45c468aa04a938bd65aa8c2c62a75321c08b4d25
S: ae38fb22d3527bdd72d639c0ba59ecc98895d514 192.168.1.129:8033
   slots: (0 slots) slave
   replicates 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 192.168.1.139:7040
>>> Send CLUSTER MEET to node 192.168.1.129:8044 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.1.139:7040.
[OK] New node added correctly.

查看所有节点信息

[root@master ~]# redis-cli -p 7020 cluster nodes
074b436c99e41aeaef4e64a7e12c5d4dde03115f 192.168.1.129:8044@18044 slave f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 0 1688392595140 3 connected
ae38fb22d3527bdd72d639c0ba59ecc98895d514 192.168.1.129:8033@18033 slave 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 0 1688392594000 2 connected
2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030@17030 master - 0 1688392596143 2 connected 5461-10922
550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022@18022 slave 45c468aa04a938bd65aa8c2c62a75321c08b4d25 0 1688392594136 1 connected
45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020@17020 myself,master - 0 1688392591000 1 connected 0-5460
f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040@17040 master - 0 1688392595000 3 connected 10923-16383

从库读写测试

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8022
192.168.1.129:8022> set sdsfd aaa
-> Redirected to slot [11393] located at 192.168.1.139:7040
OK
192.168.1.139:7040> exit
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8022 get sdsfd
"aaa"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8033 get sdsfd
"aaa"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8044 get sdsfd
"aaa"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7020 get sdsfd
"aaa"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7030 get sdsfd
"aaa"
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7040 get sdsfd
"aaa"

通过测试，我们发现redis的cluster集群不论是主库还是从库都可以进行set和get。因此，在使用中我们就没必要都去主库了。

查看主从cluster集群key的分布情况

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8022 info Keyspace
# Keyspace
db0:keys=3337,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8033 info Keyspace
# Keyspace
db0:keys=3355,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -c -p 8044 info Keyspace
# Keyspace
db0:keys=3311,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7020 info Keyspace
# Keyspace
db0:keys=3337,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7030 info Keyspace
# Keyspace
db0:keys=3355,expires=0,avg_ttl=0
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -c -p 7040 info Keyspace
# Keyspace
db0:keys=3311,expires=0,avg_ttl=0

redis集群的主从自动切换，主库挂掉后，从自动变为主

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.139 -p 7020 shutdown
[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -p 8022 cluster nodes
2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030@17030 master - 0 1688393067474 2 connected 5461-10922
f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040@17040 master - 0 1688393066468 3 connected 10923-16383
550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022@18022 myself,master - 0 1688393066000 4 connected 0-5460
074b436c99e41aeaef4e64a7e12c5d4dde03115f 192.168.1.129:8044@18044 slave f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 0 1688393068480 3 connected
ae38fb22d3527bdd72d639c0ba59ecc98895d514 192.168.1.129:8033@18033 slave 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 0 1688393066000 2 connected
45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020@17020 master,fail - 1688393002182 1688393002082 1 disconnected

手动down掉了7020端口的redis-server。从信息得知，7020挂了，8022被切换成了master

重新启动7020端口的server，再次查看

[root@master ~]# redis-server /data/redis-cluster/7020/redis.conf

[root@slave redis-4.0.10]# redis-cli -h 192.168.1.129 -p 8022 cluster nodes
2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030@17030 master - 0 1688393221386 2 connected 5461-10922
f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040@17040 master - 0 1688393222000 3 connected 10923-16383
550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022@18022 myself,master - 0 1688393220000 4 connected 0-5460
074b436c99e41aeaef4e64a7e12c5d4dde03115f 192.168.1.129:8044@18044 slave f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 0 1688393220379 3 connected
ae38fb22d3527bdd72d639c0ba59ecc98895d514 192.168.1.129:8033@18033 slave 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 0 1688393222390 2 connected
45c468aa04a938bd65aa8c2c62a75321c08b4d25 192.168.1.139:7020@17020 slave 550c98a3fa253db9db7e73c55bdebd7b99a786ca 0 1688393219377 4 connected

并没有变回来

我就想让7020做主手动切换让它成为主

要删除源文件里面源文件在会报错

[root@master ~]# rm -rf /data/redis-cluster/8001/redis.pid
[root@master ~]# rm -rf /data/redis-cluster/7020/redis.pid

[root@master ~]# rm -rf /data/redis-cluster/7020/nodes.conf

[root@master ~]# redis-trib.rb add-node --slave 192.168.1.129:8022 192.168.1.139:7020

在查看 7020 成功变回来了

[root@slave ~]# redis-cli -h 192.168.1.139 -c -p 7020 cluster failover
OK
[root@slave ~]# redis-cli -h 192.168.1.129 -p 8022 cluster nodes
074b436c99e41aeaef4e64a7e12c5d4dde03115f 192.168.1.129:8044@18044 slave f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 0 1688432935000 3 connected
ae38fb22d3527bdd72d639c0ba59ecc98895d514 192.168.1.129:8033@18033 slave 2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 0 1688432939082 2 connected
45c468aa04a938bd65aa8c2c62a75321c08b4d25 :0@0 slave,fail?,noaddr 550c98a3fa253db9db7e73c55bdebd7b99a786ca 1688430791755 1688430791750 4 disconnected
550c98a3fa253db9db7e73c55bdebd7b99a786ca 192.168.1.129:8022@18022 myself,slave a6e80af6c2f8e9a67e4cd625d3c12f8fb1ee9718 0 1688432937000 4 connected
a6e80af6c2f8e9a67e4cd625d3c12f8fb1ee9718 192.168.1.139:7020@17020 master - 0 1688432938079 5 connected 0-5460
2dcb2f5cb160ccfff92ddb8b5fa86879092a6a21 192.168.1.139:7030@17030 master - 0 1688432938000 2 connected 5461-10922
f97c2d4bd984ff9324ca79e6ff75328b7e46b17a 192.168.1.139:7040@17040 master - 0 1688432937075 3 connected 10923-16383