Operator——Redis之重启虚拟机后无法重建集群

原创

已于 2022-12-14 16:28:59 修改 · 782 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#redis #operator #cluster #重建集群 #k8s

于 2022-12-13 15:18:57 首次发布

本文记录了Redis集群在Kubernetes环境中因虚拟机重启导致集群无法重建的问题及排查过程。介绍了RedisOperator的日志信息，并给出了可能的解决方案，包括清理AOF/RDB文件等。

1. 背景介绍

由于业务需要Redis，我司选择使用Redis Operator在Kubernetes中部署一套Redis集群。

运维同学反应，当重启虚拟机之后，Redis集群无法重建集群。我把问题排查过程记录了下来，于是有了大家看到的这篇文章。

重启虚拟机之后，Redis-Operator无法重建集群，错误日志如下：

从日志信息中可以看到，Redis集群的Leader, Follower已经成功启动，但是在通过redis-cli --cluster add-node命令时出错。

# kubectl logs -f --tail 100 -n redis-system redis-operator-75f946fd68-stzjx
I1213 03:17:50.751820       1 request.go:665] Waited for 1.042141823s due to client-side throttling, not priority and fairness, request: GET:https://10.233.0.1:443/apis/events.k8s.io/v1?timeout=32s
{"level":"info","ts":1670901471.1052506,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1670901471.1055748,"logger":"setup","msg":"starting manager"}
I1213 03:17:51.105988       1 leaderelection.go:248] attempting to acquire leader lease redis-system/6cab913b.redis.opstreelabs.in...
{"level":"info","ts":1670901471.1060054,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":1670901471.1060386,"msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
I1213 03:18:08.969275       1 leaderelection.go:258] successfully acquired lease redis-system/6cab913b.redis.opstreelabs.in
{"level":"info","ts":1670901488.9695313,"logger":"controller.redis","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","source":"kind source: *v1beta1.Redis"}
{"level":"info","ts":1670901488.9696147,"logger":"controller.redis","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis"}
{"level":"info","ts":1670901488.9695785,"logger":"controller.rediscluster","msg":"Starting EventSource","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","source":"kind source: *v1beta1.RedisCluster"}
{"level":"info","ts":1670901488.9696522,"logger":"controller.rediscluster","msg":"Starting Controller","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster"}
{"level":"info","ts":1670901489.0710678,"logger":"controller.rediscluster","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"RedisCluster","worker count":1}
{"level":"info","ts":1670901489.0711362,"logger":"controller.redis","msg":"Starting workers","reconciler group":"redis.redis.opstreelabs.in","reconciler kind":"Redis","worker count":1}
{"level":"info","ts":1670901489.071197,"logger":"controllers.RedisCluster","msg":"Reconciling opstree redis Cluster controller","Request.Namespace":"redis-system","Request.Name":"redis"}
{"level":"info","ts":1670901489.0770147,"logger":"controller_redis","msg":"Redis statefulset get action was successful","Request.StatefulSet.Namespace":"redis-system","Request.StatefulSet.Name":"redis-leader"}
{"level":"info","ts":1670901489.0919821,"logger":"controller_redis","msg":"Reconciliation Complete, no Changes required.","Request.StatefulSet.Namespace":"redis-system","Request.StatefulSet.Name":"redis-leader"}
{"level":"info","ts":1670901489.0957606,"logger":"controller_redis","msg":"Redis service get action is successful","Request.Service.Namespace":"redis-system","Request.Service.Name":"redis-leader-headless"}
{"level":"info","ts":1670901489.0983348,"logger":"controller_redis","msg":"Redis service is already in-sync","Request.Service.Namespace":"redis-system","Request.Service.Name":"redis-leader-headless"}
{"