云计算基础架构（二）redis群集

最新推荐文章于 2024-08-05 02:56:50 发布

禾越

最新推荐文章于 2024-08-05 02:56:50 发布

阅读量566

点赞数

文章标签： redis 云计算缓存

本文链接：https://blog.youkuaiyun.com/qq_57258570/article/details/123180069

版权

本文档详细介绍了如何在Redis 3.2.12版本上搭建三主三从的集群环境。首先，通过安装和配置Redis服务，确保各节点配置文件中的IP地址正确。接着，使用redis-trib.rb工具创建集群，并处理因密码问题导致的连接错误。集群创建成功后，通过检查节点状态和分配槽位，验证集群的正常运行。此外，还展示了如何在集群中添加、删除节点以及进行数据操作和分布测试。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

环境:
redis版本: redis-3.2.12

实现: 3主 3从
   node01: 1主1从 (主从千万不要部署在一起单机单服务 (nginx php mysql python tomcat)
   node02: 1主1从
   node03: 1主1从

一. 基础环境的安装
1.安装redis 所有节点(node01 node02 node03)
mkdir -pv /etc/redis-cluster
yum install redis -y # gw extra

2.安装集群相关软件包
yum install ruby ruby-devel rubygems rpm-build -y

# 3.修改系统参数 (PXE装机时已经优化)
# echo "session required /lib64/security/pam_limits.so" >> /etc/pam.d/login
# wget http://10.15.200.8/sa_utils/21-nofile.conf -O /etc/security/limits.d/21-nofile.conf

# Increased maximum number of open files to 100032 (it was originally set to 1024)
# 内核优化: vm.overcommit_memory = 1' to /etc/sysctl.conf
# 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'

# 4.修改配置文件将node01的配置文件分别scp到node02 node03
# /etc/redis-cluster/nodes_3000.conf /etc/redis-cluster/nodes_3001.conf
# 然后修改配置文件:
# bind 10.15.200.101 127.0.0.1 # 只修改 10.15.200.101 将此ip修改为当前node节点对应的ip地址
# [root@node01 ~]# scp /etc/redis-cluster/{nodes_3000.conf,nodes_3001.conf} node02:/etc/redis-cluster/
# [root@node01 ~]# scp /etc/redis-cluster/{nodes_3000.conf,nodes_3001.conf} node03:/etc/redis-cluster/
# [root@node02 ~]# sed -i 's/10.15.200.101/10.15.200.102/g' /etc/redis-cluster/*.conf
# [root@node03 ~]# sed -i 's/10.15.200.101/10.15.200.103/g' /etc/redis-cluster/*.conf

node01:
wget http://10.15.200.8/files/redis_cluster/node01_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
wget http://10.15.200.8/files/redis_cluster/node01_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.conf

node02:
wget http://10.15.200.8/files/redis_cluster/node02_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
wget http://10.15.200.8/files/redis_cluster/node02_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.conf

node03:
wget http://10.15.200.8/files/redis_cluster/node03_nodes_3000.conf -O /etc/redis-cluster/nodes_3000.conf
wget http://10.15.200.8/files/redis_cluster/node03_nodes_3001.conf -O /etc/redis-cluster/nodes_3001.conf

日志文件: tailf /var/log/redis/3000.log
如果不修改IP地址报如下的错误:
17491:M 28 Feb 08:46:11.156 # Creating Server TCP listening socket 10.15.200.101:3000: bind: Cannot assign requested address

正常的日志, 有如下的提示:
17608:M 28 Feb 08:48:19.287 * The server is now ready to accept connections on port 3000

5.启动服务 (三个节点: node01 node02 node03 都要运行如下两个命令启动3000 3001的服务:)
/usr/bin/redis-server /etc/redis-cluster/nodes_3000.conf
# tailf /var/log/redis/3000.log

/usr/bin/redis-server /etc/redis-cluster/nodes_3001.conf
# tailf /var/log/redis/3001.log

6.查看日志或查看进程 (以3000端口号为例其它同理)
1) [root@node01 ~]# tailf /var/log/redis/3000.log
2) [root@node01 ~]# ps -ef | grep 3000

7.服务全部重启后再次查看端口号 (校验所有的服务器的redis是否全部启动)
[root@node01 ~]# netstat -tulanp | grep :300
tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN 19899/redis-server
tcp 0 0 10.15.200.101:3000 0.0.0.0:* LISTEN 19899/redis-server
tcp 0 0 127.0.0.1:3001 0.0.0.0:* LISTEN 20282/redis-server
tcp 0 0 10.15.200.101:3001 0.0.0.0:* LISTEN 20282/redis-server

# 端口号 3000 13000 3001 13001 (含义)
8.
[root@node01 ~]# cat /etc/redis-cluster/node_3000.conf
47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 :0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0
[root@node01 ~]# cat /etc/redis-cluster/node_3001.conf
947d6460c205fdb3a12445f5c442e57330b0560c :0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0

二. 配置集群Cluster
# [root@node01 ~]# gem install redis
# Fetching: redis-4.2.5.gem (100%)
# ERROR: Error installing redis:
redis requires Ruby version >= 2.3.0.

# https://rubygems.org/gems/redis/versions/3.2.2
# wget https://rubygems.org/downloads/redis-3.2.2.gem /usr/local/src

wget http://10.15.200.8/sa_utils/redis-3.2.2.gem -O /usr/local/src/redis-3.2.2.gem

[root@node01 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
Successfully installed redis-3.2.2
Parsing documentation for redis-3.2.2
Installing ri documentation for redis-3.2.2
1 gem installed

node02 node03
wget http://10.15.200.8/sa_utils/redis-3.2.2.gem -O /usr/local/src/redis-3.2.2.gem

[root@node02 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
Successfully installed redis-3.2.2
Parsing documentation for redis-3.2.2
Installing ri documentation for redis-3.2.2
1 gem installed

[root@node03 ~]# gem install -l /usr/local/src/redis-3.2.2.gem
Successfully installed redis-3.2.2
Parsing documentation for redis-3.2.2
Installing ri documentation for redis-3.2.2
1 gem installed

[root@node01 ~]# wget http://10.15.200.8/packages/redis-3.2.12.tar.gz -O /usr/local/src/redis-3.2.12.tar.gz
[root@node01 ~]# cd /usr/local/src/ && tar -zxvf redis-3.2.12.tar.gz && cp redis-3.2.12/src/redis-trib.rb /usr/local/sbin
[root@node01 src]# /usr/local/sbin/redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>

创建集群: replicas表示每个master需要有几个slave

[root@node01 ~]# /usr/local/sbin/redis-trib.rb create --replicas 1 10.15.200.101:3000 10.15.200.101:3001 10.15.200.102:3000 10.15.200.102:3001 10.15.200.103:3000 10.15.200.103:3001

--replicas 1: 为集群中的每个主节点创建一个从节点
集群正常工作至少需要3个主节点: 以上是三主三从

>>> Creating cluster
[ERR] Sorry, can't connect to node 10.15.200.101:3000

[root@node01 ~]# vim /usr/local/share/gems/gems/redis-3.2.2/lib/redis/client.rb +16
8 DEFAULTS = {
9 :url => lambda { ENV["REDIS_URL"] },
10 :scheme => "redis",
11 :host => "127.0.0.1",
12 :port => 6379,
13 :path => nil,
14 :timeout => 5.0,
15 :connect_timeout => 5.0,
16 :password => 'admin', # 修改此处的密码 (一定要加引号)

[root@node01 ~]# /usr/local/sbin/redis-trib.rb create --replicas 1 10.15.200.101:3000 10.15.200.101:3001 10.15.200.102:3000 10.15.200.102:3001 10.15.200.103:3000 10.15.200.103:3001

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.15.200.101:3000
10.15.200.102:3000
10.15.200.103:3000
Adding replica 10.15.200.102:3001 to 10.15.200.101:3000
Adding replica 10.15.200.101:3001 to 10.15.200.102:3000
Adding replica 10.15.200.103:3001 to 10.15.200.103:3000
M: 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000
slots:0-5460 (5461 slots) master
S: 947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001
replicates 24f84465fdd543dd8f6a9e50375afd793db7941b
M: 24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000
slots:5461-10922 (5462 slots) master
S: 08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001
replicates 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75
M: d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000
slots:10923-16383 (5461 slots) master
S: d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001
replicates d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.....
>>> Performing Cluster Check (using node 10.15.200.101:3000)
M: 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001
slots: (0 slots) slave
replicates 24f84465fdd543dd8f6a9e50375afd793db7941b
S: 08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001
slots: (0 slots) slave
replicates 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75
S: d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001
slots: (0 slots) slave
replicates d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11
M: d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

[root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
10.15.200.103:3000> cluster nodes
08e742f053d23c8f1d2ed8d5dd01d6dca80c9d6e 10.15.200.102:3001 slave 47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 0 1614518465535 4 connected
947d6460c205fdb3a12445f5c442e57330b0560c 10.15.200.101:3001 slave 24f84465fdd543dd8f6a9e50375afd793db7941b 0 1614518464528 3 connected
47a9332d84fbe0ba44942e47722b3cf6fa1f2c75 10.15.200.101:3000 master - 0 1614518464023 1 connected 0-5460
d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 10.15.200.103:3000 myself,master - 0 0 5 connected 10923-16383
d5f8e89d6ecd483751991d201e1ed689cc04a28a 10.15.200.103:3001 slave d5bbda1bfeb49d708a1c415c8e0e06fd3969ff11 0 1614518461507 6 connected
24f84465fdd543dd8f6a9e50375afd793db7941b 10.15.200.102:3000 master - 0 1614518463520 3 connected 5461-10922

10.15.200.103:3000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:5
cluster_stats_messages_sent:7209
cluster_stats_messages_received:7209

三. 测试

1）get 和 set数据
[root@node01 ~]# echo 'hello_world' | /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin' -x set mykey
OK

[root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
10.15.200.101:3000> get mykey
-> Redirected to slot [14687] located at 10.15.200.103:3000
"hello_world\n" # 直接根据hash匹配切换到相应的slot的节点上

# 将 10.15.200.103 3000 端口对应的服务停止之后会重分布
[root@node01 ~]# /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin'
10.15.200.101:3000> get mykey
-> Redirected to slot [14687] located at 10.15.200.103:3001
"hello_world\n"

写入一定数量的数据: # node01 直接命令行粘贴
for i in `seq 1 10000`
do
echo "hello_world_$i" | /usr/bin/redis-cli -c -p 3000 -h 10.15.200.101 -a 'admin' -x set "mykey_$i"
done

查看现集群分布情况:
[root@node01 ~]# /usr/local/sbin/redis-trib.rb info 10.15.200.101:3000
10.15.200.103:3001 (bbf0844e...) -> 3351 keys | 5461 slots | 1 slaves.
10.15.200.102:3000 (462e7cf4...) -> 3316 keys | 5462 slots | 1 slaves.
10.15.200.102:3001 (6ac1063d...) -> 3335 keys | 5461 slots | 1 slaves.
[OK] 10002 keys in 3 masters.
0.61 keys per slot on average.

增加一个主节点:
注意: 最后面一段地址+端口号: 10.15.200.101:3000 是现集群的任意一个正常的IP:PORT都可以

[root@node01 ~]# /usr/local/sbin/redis-trib.rb add-node 10.15.200.104:3000 10.15.200.101:3000
>>> Adding node 10.15.200.104:3000 to cluster 10.15.200.101:3000
>>> Performing Cluster Check (using node 10.15.200.101:3000)
S: 0d0c41cfa78c4372dc64da8ac957d5344ef69424 10.15.200.101:3000
slots: (0 slots) slave
replicates 6ac1063d0e014e89606a84c0cece3ddf4ee6b68e
M: bbf0844ee9b5c25635dc8f42f674e4ce631a64f9 10.15.200.103:3001
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 86b188db102f856a129cafda55c7f6da57982881 10.15.200.101:3001
slots: (0 slots) slave
replicates 462e7cf49b2e9e1159ffe94439cd0afc231e9f4c
M: 462e7cf49b2e9e1159ffe94439cd0afc231e9f4c 10.15.200.102:3000
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 107b83d08c63eed219242f7a32585d626ed47dba 10.15.200.103:3000
slots: (0 slots) slave
replicates bbf0844ee9b5c25635dc8f42f674e4ce631a64f9
M: 6ac1063d0e014e89606a84c0cece3ddf4ee6b68e 10.15.200.102:3001
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.15.200.104:3000 to make it join the cluster.
[OK] New node added correctly. # 添加成功

[root@node01 ~]# /usr/local/sbin/redis-trib.rb check 10.15.200.104:3000 # 检查集群状态
>>> Performing Cluster Check (using node 10.15.200.104:3000)
M: e140c57f1eb4d1694c14f05b2bafc998e00fbc94 10.15.200.104:3000
slots: (0 slots) master # 默认刚刚新增的主节点是没有slots的, 这样在存取数据的时候不会被选择
0 additional replica(s)

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

# 查看key的分部情况
[root@node01 ~]# /usr/local/sbin/redis-trib.rb info 10.15.200.101:3000
10.15.200.104:3000 (e140c57f...) -> 0 keys | 0 slots | 0 slaves.
10.15.200.103:3001 (bbf0844e...) -> 3351 keys | 5461 slots | 1 slaves.
10.15.200.102:3000 (462e7cf4...) -> 3316 keys | 5462 slots | 1 slaves.
10.15.200.102:3001 (6ac1063d...) -> 3335 keys | 5461 slots | 1 slaves.
[OK] 10002 keys in 4 masters.
0.61 keys per slot on average.

[root@node01 ~]# /usr/local/sbin/redis-trib.rb reshard 10.15.200.104:3000
>>> Performing Cluster Check (using node 10.15.200.104:3000)
M: e140c57f1eb4d1694c14f05b2bafc998e00fbc94 10.15.200.104:3000
slots: (0 slots) master
0 additional replica(s)

删除节点:
[root@node01 ~]# /usr/local/sbin/redis-trib.rb check 10.15.200.104:3000
>>> Performing Cluster Check (using node 10.15.200.104:3000)
M: 4319c3a09b885fc0c818c7db946293f46ec8a0b9 10.15.200.104:3000 # 注意此处的 ID号后面删除时会用到

[root@node01 ~]# /usr/local/sbin/redis-trib.rb del-node 10.15.200.104:3000 4319c3a09b885fc0c818c7db946293f46ec8a0b9
>>> Removing node 4319c3a09b885fc0c818c7db946293f46ec8a0b9 from cluster 10.15.200.104:3000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

报错:
[ERR] Node 10.15.200.104:3000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

由于服务器重启原因, 导致该节点中默认生成的配置或历史存储数据不一致导致启动集群报错, 需要清除该节点的文件, 必要时清除数据库, 步骤如下:
[root@node04 ~]# rm -fr /var/lib/redis/*
[root@node04 ~]# redis-cli -h 127.0.0.1 -a 'admin' -p 3000

3.关闭redis服务
ps -ef
kill -9
4.启动服务
/usr/bin/redis-server /etc/redis-cluster/nodes_3000.conf

[ERR] Calling MIGRATE: ERR Target instance replied with error: NOAUTH Authentication required.