最近,在备份Elasticsearch集群上发现其snapshot的restore功能失灵了,一直无法restore主集群的snapshot,每次都在调用创建repository API的时候失败。GET ~/_snapshot API返回如下的内容。而通过Azure Explorer检查相应的Azure Storage Account,snapshot-0401-72d8早已经被清除了。
{
"auto-restore-repo" : {
"type" : "azure",
"settings" : {
"container" : "snapshot-0401-72d8",
"location_mode" : "secondary_only",
"compress" : "true",
"account" : "elasticsearch"
}
}
}
进一步检查Elasticsearch的日志,发现了下面相关错误信息:
[2016-04-12 00:02:29,526][DEBUG][cluster.service ] [ESNode-IN_0] processing [putsitory [auto-restore-repo]]: execute
[2016-04-12 00:02:29,526][WARN ][repositories ] [ESNode-IN_0] failed to create repository [auto-restore-repo]
org.elasticsearch.ElasticsearchIllegalStateException: trying to modify or unregister repository that is currently used
at org.elasticsearch.repositories.RepositoriesService.ensureRepositoryNotInUse(RepositoriesService.java:420)
at org.elasticsearch.repositories.RepositoriesService.access$000(RepositoriesService.java:55)
at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:108)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:374)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:204)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:167)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)