当前telemetry内部的数据协同是通过tooz, tooz默认时使用redis,经常出现的是重启环境后,redis异常,redis异常日志:
9317:M 07 May 10:35:33.120 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
9317:M 07 May 10:35:33.120 # Server started, Redis version 3.0.6
9317:M 07 May 10:35:33.120 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1'
for this to take effect.
9317:M 07 May 10:35:33.121 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'
as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
9317:M 07 May 10:35:34.682 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
修复方式:
cd /var/lib/redis
cp appendonly.aof appendonly.aof.bak
redis-check-aof --fix appendonly.aof
systemctl start redis.service
即可