ubuntu14.04 redis集群部署

一、redis安装

sudo apt-get remove redis-server --purge   # 删除旧版
# apt-get autoremove
sudo apt-get install -y python-software-properties
sudo apt-get install software-properties-common
# sudo add-apt-repository ppa:chris-lea/redis-server
sudo add-apt-repository -y ppa:rwky/redis
sudo apt-get update
sudo apt-get install -y redis-server

二、创建六个节点，三个master，三个slave

sudo mkdir /usr/local/redis_cluster
cd /usr/local/redis_cluster
sudo mkdir 7000 7001 7002 7003 7004 7005
# sudo mkdir /usr/local/redis_cluster/{7000,7001,7002,7003,7004,7005}
sudo cp -p /etc/redis/redis.conf /usr/local/redis_cluster/7000/    # 7001、7002、7003、7004、7005也是一样

• 修改配置7000/redis.conf，其他节点也是一样

bind 0.0.0.0 ::1                                //允许其他ip访问
pidfile /var/run/redis/redis-server-7000.pid    //pidfile文件对应7000,7002,7003
port  7000                                     //端口7000,7002,7003
logfile /var/log/redis/redis-server-7000.log    //日志文件
dbfilename dump-7000.rdb                       //文件保存
cluster-enabled  yes                           //开启集群  把注释#去掉
cluster-config-file  nodes-7000.conf           //集群的配置  配置文件首次启动自动生成 7000,7001,7002
cluster-node-timeout  5000                     //请求超时  设置5秒够了
appendonly  yes                                //aof日志开启  有需要就开启，它会每次写操作都记录一条日志

三、使用redis用户启动

• 复制redis执行文件

sudo cp /etc/init.d/redis-server /etc/init.d/redis-server-7000

• 修改redis-server-7000配置文件对应位置

DAEMON=/usr/bin/redis-server
DAEMON_ARGS=/usr/local/redis_cluster/7000/redis.conf
NAME=redis-server
DESC=redis-server-7000

RUNDIR=/var/run/redis
PIDFILE=$RUNDIR/redis-server-7000.pid

• 启动redis-server-7000

sudo service redis-server-7000 {start|stop|restart|force-reload|status}

• 开机启动设置

sudo update-rc.d redis-server-7000 defaults 20    # 开机启动

• 其他节点

cd /usr/local/redis_cluster
sudo cp -p 7000/redis.conf 7001/
sudo vim 7001/redis.conf    # 全局替换7000改成70001

cd /etc/init.d
sudo cp redis-server-7000 redis-server-7001
sudo vim redis-server-7001    # 全局替换7000改成70001

sudo service redis-server-7001 {start|stop|restart|force-reload|status}
sudo update-rc.d redis-server-7001 defaults 20    # 开机启动

• ps axu | grep redis #查看是否启动成功
  
• sudo netstat -tnlp | grep redis #可以看到redis监听端口
  

四、直接使用root用户启动

touch run_cluster_redis.sh

# redis-server /usr/local/redis_cluster/7000/redis.conf
# redis-server /usr/local/redis_cluster/7001/redis.conf
# redis-server /usr/local/redis_cluster/7002/redis.conf
# redis-server /usr/local/redis_cluster/7003/redis.conf
# redis-server /usr/local/redis_cluster/7004/redis.conf
# redis-server /usr/local/redis_cluster/7005/redis.conf
service redis-server-7000 $1
service redis-server-7001 $1
service redis-server-7002 $1
service redis-server-7003 $1
service redis-server-7004 $1
service redis-server-7005 $1
chmod +x run_cluster_redis.sh
sudo ./run_cluster_redis.sh {start|stop|restart|force-reload|status}

五、创建集群

• redis5.x版本中，redis-cli --cluster命令替代./redis-trib.rb命令

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
--cluster-replicas 1
# 允许外部访问，使用本机内网ip地址替换127.0.0.1
# redis-cli --cluster create 172.31.1.1:7000 172.31.1.1:7001 \
# 172.31.1.1:7002 172.31.1.1:7003 172.31.1.1:7004 172.31.1.1:7005 \
# --cluster-replicas 1

• 查看节点信息

redis-cli -p 7000 cluster nodes

• 使用集群添加数据
  

六、集群命令

• 添加节点

# 第一个参数是新节点的地址，第二个参数是任意一个已经存在的节点的IP和端口. 
redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

• 删除节点

# 第一个参数是任意一个节点的地址,第二个节点是你想要移除的节点地址。
# 使用同样的方法移除主节点,不过在移除主节点前，需要确保这个主节点是空的. 如果不是空的,需要将这个节点的数据重新分片到其他主节点上.
redis-cli --cluster del-node 127.0.0.1:7000 `<node-id>`

# 删除7000节点
jtserver@ip-172-31-6-225:~$ redis-cli --cluster del-node 127.0.0.1:7001 d9955463de9bd9e8b79a923defde4f3885ab19a6
>>> Removing node d9955463de9bd9e8b79a923defde4f3885ab19a6 from cluster 127.0.0.1:7001
[ERR] Node 127.0.0.1:7000 is not empty! Reshard data away and try again.

# 删除7006节点
jtserver@ip-172-31-6-225:~$ redis-cli --cluster del-node 127.0.0.1:7001 9159f14fcd50f83b85ed4704d64e73ef8d253661
>>> Removing node 9159f14fcd50f83b85ed4704d64e73ef8d253661 from cluster 127.0.0.1:7001
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

• 重新分片

redis-cli --cluster reshard 127.0.0.1:7000
redis-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes

• all和done两种模式

redis-cli --cluster reshard 127.0.0.1:7000
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 100
What is the receiving node ID? 166fe530d89ba851b540083adb0463bb77c51ae6
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
# 使用all， 获取100slot，从其他节点均摊
Source node #1: all

# 使用done，选择要均摊的节点
Source node #1: d9955463de9bd9e8b79a923defde4f3885ab19a6
Source node #2: done

# 从一个节点转移100个slot到另外一个节点
redis-cli --cluster reshard 127.0.0.1:7000  --cluster-from d9955463de9bd9e8b79a923defde4f3885ab19a6  --cluster-to 166fe530d89ba851b540083adb0463bb77c51ae6  --cluster-slots 100 --cluster-yes

• 其他命令

jtserver@ip-172-31-6-225:~$ redis-cli --cluster help
Cluster Manager Commands:
  create         host1:port1 ... hostN:portN
                 --cluster-replicas <arg>
  check          host:port
  info           host:port
  fix            host:port
  reshard        host:port
                 --cluster-from <arg>
                 --cluster-to <arg>
                 --cluster-slots <arg>
                 --cluster-yes
                 --cluster-timeout <arg>
                 --cluster-pipeline <arg>
  rebalance      host:port
                 --cluster-weight <node1=w1...nodeN=wN>
                 --cluster-use-empty-masters
                 --cluster-timeout <arg>
                 --cluster-simulate
                 --cluster-pipeline <arg>
                 --cluster-threshold <arg>
  add-node       new_host:new_port existing_host:existing_port
                 --cluster-slave
                 --cluster-master-id <arg>
  del-node       host:port node_id
  call           host:port command arg arg .. arg
  set-timeout    host:port milliseconds
  import         host:port
                 --cluster-from <arg>
                 --cluster-copy
                 --cluster-replace
  help

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

七、重置集群

• 停止所有节点，使用命令杀死进程

 ps axu | grep redis | grep cluster | awk '{print $2}' | xargs sudo kill -9

• 删除数据文件、节点文件、aof日志和日志文件

sudo rm -rf /var/lib/redis/dump-*
sudo rm -rf /var/lib/redis/nodes-*
sudo rm -rf /var/lib/redis/appendonly.aof
sudo rm -rf /var/log/redis/redis-server-*

• 重启服务，再按照上述创建集群

sudo service redis-server-(7000|7001|7002|7003|7004|7005) start

八、问题

• Node 127.0.0.1:7000 has slots in importing state 12182.

redis-cli --cluster fix 127.0.0.1:7000

• [ERR] Node 127.0.0.1:7006 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

# Keyspace
jtserver@ip-172-31-6-225:~$ redis-cli -p 7006
127.0.0.1:7006> info
db0:keys=1,expires=0,avg_ttl=0
127.0.0.1:7006> KEYS *
1) "foo"
127.0.0.1:7006> get foo
(error) CLUSTERDOWN The cluster is down
127.0.0.1:7006> flushall
OK

九、参考

• redis官方集群英文指导
• redis官方集群中文指导
• https://www.cnblogs.com/cjsblog/p/9048545.html
• https://www.cnblogs.com/lykxqhh/p/5690923.html
• redis cluster管理工具redis-trib.rb详解