spark 连接 ES java.net.SocketTimeoutException 异常

INFO - Subtask: INFO - Exception in thread "main" java.net.SocketTimeoutException
INFO - Subtask: INFO - at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
INFO - Subtask: INFO - at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
INFO - Subtask: INFO - at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
INFO - Subtask: INFO - at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
INFO - Subtask: INFO - at java.lang.Thread.run(Thread.java:748)
INFO - Subtask: INFO - Command exited with return code 1
ERROR - Bash command failed

• 问题并不是必现，ES 压力大的时候经常出现

• 原因：ES shard 数过多导致(nodes=18, index=3K+, shards=3W+)

• 解决：部分 index 数据量很小，但是默认 使用 5 个 shard ，以及一个备份，他们占用了大部分的 shard 数，通过合并、削减，降低至 ( index=1K+, shards=1W+) 后不再出现

• 尽量保证单个 shard 不超过 5G ，但也不要过多