Elasticsearch集群Red,分片无法恢复:cannot allocatebecause all found copies of the shard are either stale or c

问题

在多次重启节点时,主副分片0出现了无法分配的问题。

elasticsearch 使用 Allocation IDs 的概念,这是区分不同分片的唯一标识(UUIDS)。

Allocation IDs存储在 shard 级元信息中,每个 shard 都有自己的Allocation ID,同时集群级元信息中记录了一个被认为是最新shard 的Allocation ID集合,这个集合称为 in-sync allocation IDs。
如果由于网络或者其他原因,主副shard没有同步,那么副本的shard会将被从in-sync allocation IDs踢出。

猜想可能是出现了上述的问题。

解决

使用下面的命令进行查看:

GET /_cluster/allocation/explain

explain API 告诉我们为什么主分片处于未分配状态,同时还提供了基于每个节点上的更详细的分配信息。主节点在集群当前可用节点中无法找到同步的(in-sync)分片副本。

{
  "index" : "my_index",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2017-01-26T09:20:24.054Z",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt",
  "node_allocation_decisions" : [
    {
      "node_id" : "0ZJS1ffpRg-E_5L0Q0NnDA",
      "node_name" : "data5-3",
      "transport_address" : "192.168.1.108:9303",
      "node_decision" : "no",
      "store" : {
          "found": false
      }
    },
     {
      "node_id" : "0ZJS1ffpRg-E_5L0Q0NnDA",
      "node_name" : "data5-3",
      "transport_address" : "192.168.1.108:9303",
      "node_decision" : "no",
      "store" : {
          "found": false
      }
    },
    .......
    .......
    {
      "node_id" : "JSofMo_0TSieKnfI4HmEcQ",
      "node_name" : "data6",
      "transport_address" : "192.168.1.109:9300",
      "node_decision" : "no",
      "store" : {
        "in_sync" : false,
        "allocation_id" : "J3gygIvYT06tKSW5rYkGtw"
      }
    }
  ]
}

cannot allocate because all found copies of the shard are either stale or corrupt
从上面的结果中,可以看到在data6节点上可用的分片副本是陈旧的( store.in_sync = false )。启动拥有in-sync 分片副本的那个节点将使集群重新变为 green。如果那个节点永远都回来了呢?

reroute API 提供了一个子命令 allocate_stale_primary ,用于将一个陈旧的分片分配为主分片。使用此命令意味着丢失给定分片副本中缺少的数据。如果同步分片只是暂时不可用,使用此命令意味着在同步副本中最近更新的数据。应该把它看作是使群集至少运行一些数据的最后一种措施。在所有分片副本都不存在的情况下,还可以强制Elasticsearch使用空分片副本分配主分片,这意味着丢失与该分片相关联的所有先前数据。 不言而喻,allocate_empty_primary 命令只能用于最糟糕的情况,其含义很好理解。

使用命令进行恢复:

POST _cluster/reroute?pretty
{
	"commands": [
		{
			"allocate_stale_primary": {
				"index": "my_index",
				"shard": 0,
				"node": "data6",
				"accept_data_loss": true
			}
		}
	]
}
[root@mobilexw ~]# curl -XGET localhost:9200/_cluster/allocation/explain {"note":"No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.","index":"pds_dacfga_egebpl","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"ALLOCATION_FAILED","at":"2025-03-02T16:10:26.604Z","failed_allocation_attempts":1,"details":"failed shard on node [4bMI2hXVThm4Qr1axNbCjA]: shard failure, reason [merge failed], failure MergeException[org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=77ca06fc actual=dc58566c (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/opt/dpmon/server/vendor/elasticsearch/data/nodes/0/indices/uiziMJtlSXWQ4SCFdQ8zPQ/0/index/_39qu.cfs\") [slice=_39qu_Lucene80_0.dvd]))]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=77ca06fc actual=dc58566c (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/opt/dpmon/server/vendor/elasticsearch/data/nodes/0/indices/uiziMJtlSXWQ4SCFdQ8zPQ/0/index/_39qu.cfs\") [slice=_39qu_Lucene80_0.dvd]))]; ","last_allocation_status":"no_valid_shard_copy"},"can_allocate":"no_valid_shard_copy","allocate_explanation":"cannot allocate because all found copies of the shard are either stale or corrupt","node_allocation_decisions":[{"node_id":"4bMI2hXVThm4Qr1axNbCjA","node_name":"node-1","transport_address":"10.39.0.162:9300","node_attributes":{"ml.machine_memory":"67387416576","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"512","ml.max_jvm_size":"4294967296"},"node_decision":"no","store":{"in_sync":true,"allocation_id":"u37eokDMRGyGni3c6xnwdw","store_exception":{"type":"corrupt_index_exception","reason":"failed engine (reason: [merge failed]) (resource=preexisting_corruption)","caused_by":{"type":"i_o_exception","reason":"failed engine (reason: [merge failed])","caused_by":{"type":"corrupt_index_exception","reason":"checksum failed (hardware problem?) : expected=77ca06fc actual=dc58566c (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/opt/dpmon/server/vendor/elasticsearch/data/nodes/0/indices/uiziMJtlSXWQ4SCFdQ8zPQ/0/index/_39qu.cfs\") [slice=_39qu_Lucene80_0.dvd]))"}}}}}]}[root@mobilexw ~]# 这个报错里面有pds_dacfga_egebpl这个索引的信息吗?
03-27
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值