Kuboard突然无法访问提示：Failed to connect to the database

原创已于 2025-08-06 10:52:18 修改 · 437 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#数据库 #java #服务器

于 2025-08-06 10:42:09 首次发布

一、问题，kuboard访问报错

{
  "message": "Failed to connect to the database.",
  "type": "Internal Server Error"
}

二、排查

查看kuboard运行情况

docker ps | grep kuboard

查看kuboard容器日志

docker logs -f  --tail=10  容器ID

[root@nb003 ~]# docker logs -f  --tail=10  xx
time="2023-09-23T05:15:08Z" level=error msg="failed to rotate keys: etcdserver: mvcc: database space exceeded"
{"level":"warn","ts":"2023-09-23T13:15:12.504+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-8f19b170-257f-4a30-942f-1be1122e3be0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
time="2023-09-23T05:15:12Z" level=error msg="Storage health check failed: create auth request: etcdserver: mvcc: database space exceeded"

日志如上，发现提示ResourceExhausted desc = etcdserver: mvcc: database space exceeded，这表示etcd服务磁盘空间不足了，默认的空间配额限制为2G，超出空间配额限制就会影响服务，所以需要定期清理。
故查看数据映射的空间大小，找到自己的kuboard-data，查看
etcd db占用空间大小，发现已经达到默认的空间配额限制为2G的最大值。

[root@nb003 snap]# ls -lrth
总用量 2.1G
-rw-r--r-- 1 root root 8.0K 9月  19 11:49 0000000000000005-00000000005c4542.snap
-rw-r--r-- 1 root root 8.0K 9月  20 08:41 0000000000000005-00000000005c6c53.snap
-rw-r--r-- 1 root root 8.0K 9月  21 05:33 0000000000000005-00000000005c9364.snap
-rw-r--r-- 1 root root 8.0K 9月  22 02:26 0000000000000005-00000000005cba75.snap
-rw-r--r-- 1 root root 8.0K 9月  23 07:13 0000000000000005-00000000005ce186.snap
-rw------- 1 root root 2.0G 9月  23 11:57 db
————————————————

进入kuboard容器内部，查看etcd的情况：可以看到在ERRORS列里同样也提示了一个警告alarm:NOSPACE空间不足

[root@nb003 snap]# docker exec -it a2caf8010e75 bash
root@a2caf8010e75:/# ETCDCTL_API=3 etcdctl --endpoints="http://127.0.0.1:2379" --write-out=table endpoint status
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
|       ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX |             ERRORS             |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
| http://127.0.0.1:2379 | 59a9c584ea2c3f35 |  3.4.14 |  2.1 GB |      true |      false |         6 |    6089300 |            6089300 |   memberID:6460912315094810421 |
|                       |                  |         |         |           |            |           |            |                    |                 alarm:NOSPACE  |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+

三、解决办法

在kuboard容器中依次做如下操作：

# 备份db
etcdctl snapshot save backup.db
# 查看当前版本
rev=$(ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
# 压缩旧版本
ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 compact $rev
# 整理多余的空间
ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 defrag
# 取消告警信息（之前有nospace的告警）
ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 alarm disarm
# 再次查看etcd的状态（发现ERROR字段已为空）
ETCDCTL_API=3 etcdctl --endpoints="http://127.0.0.1:2379" --write-out=table endpoint status

过程

root@a2caf8010e75:/# etcdctl snapshot save backup.db
{"level":"info","ts":1695447648.315712,"caller":"snapshot/v3_snapshot.go:119","msg":"created temporary db file","path":"backup.db.part"}
{"level":"info","ts":"2023-09-23T13:40:48.317+0800","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":1695447648.3172774,"caller":"snapshot/v3_snapshot.go:127","msg":"fetching snapshot","endpoint":"127.0.0.1:2379"}
{"level":"info","ts":"2023-09-23T13:41:03.646+0800","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
{"level":"info","ts":1695447663.8131642,"caller":"snapshot/v3_snapshot.go:142","msg":"fetched snapshot","endpoint":"127.0.0.1:2379","size":"2.1 GB","took":15.497392681}
{"level":"info","ts":1695447663.8132935,"caller":"snapshot/v3_snapshot.go:152","msg":"saved","path":"backup.db"}
Snapshot saved at backup.db
root@a2caf8010e75:/# rev=$(ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
root@a2caf8010e75:/#  ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 compact $rev
compacted revision 6077603
root@a2caf8010e75:/#  ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 defrag
Finished defragmenting etcd member[http://127.0.0.1:2379]
root@a2caf8010e75:/# ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 alarm disarm
memberID:6460912315094810421 alarm:NOSPACE 
root@a2caf8010e75:/# ETCDCTL_API=3 etcdctl --endpoints="http://127.0.0.1:2379" --write-out=table endpoint status
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://127.0.0.1:2379 | 59a9c584ea2c3f35 |  3.4.14 |  127 kB |      true |      false |         6 |    6089454 |            6089454 |        |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
root@a2caf8010e75:/#

四、在 ETCD 启动命令中添加下面两个参数：

# 表示每隔一个小时自动压缩一次
--auto-compaction-retention=1
# 磁盘空间调整为 8G，官方建议最大 8G（单位是字节）
--quota-backend-bytes=8388608000