kibana 报错 elasticsearch service Unavailable

最新推荐文章于 2025-07-01 14:23:15 发布

转载最新推荐文章于 2025-07-01 14:23:15 发布 · 5.7k 阅读

1 ·

CC 4.0 BY-SA版权

原文链接：https://yq.aliyun.com/articles/619152

文章标签：

#大数据 #java

本文记录了一次Elasticsearch服务出现故障的过程，包括Kibana提示的错误信息和服务不可用的状态变化，通过查看配置文件及日志定位问题所在，并详细记录了故障排除步骤。

今天客户反映后台管理系统审核功能失效, 到kibana查看错误信息时发现, kibana提示elasticsearch: service Unavailable, 如下:

ba236295f878d5b72886cd03f495754325617b02

到kibana查看日志信息如下:

{"type":"log","@timestamp":"2018-06-29T23:08:09+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19806,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - [master_not_discovered_exception] null","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-06-29T23:08:11+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19806,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - Service Unavailable","prevState":"red","prevMsg":"[master_not_discovered_exception] null"}
{"type":"log","@timestamp":"2018-06-29T23:45:20+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19806,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - Request Timeout after 30000ms","prevState":"red","prevMsg":"Service Unavailable"}
{"type":"log","@timestamp":"2018-06-29T23:48:43+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19806,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - [master_not_discovered_exception] null","prevState":"red","prevMsg":"Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2018-06-29T23:48:46+00:00","tags":["status","plugin:elasticsearch","error"],"pid":19806,"name":"plugin:elasticsearch","state":"red","message":"Status changed from red to red - Service Unavailable","prevState":"red","prevMsg":"[master_not_discovered_exception] null"}

提示elasticsearch组件请求错误.

查看kibana配置:

# The maximum payload size in bytes on incoming server requests.
# server.maxPayloadBytes: 1048576

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://**.**.75.206:9200"

找到elasticsearch服务所在IP, 到elasticsearch机器上查看其配置:

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /server/pd/elasticsearch-2.3.3/data
#
# Path to log files:
#
path.logs: /server/pd/elasticsearch-2.3.3/logs

#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["**.**.52.15:9300", "**.**.81.194:9300"]

#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 3

获取elasticsearch节点地址, 查看path.logs配置地址下的日志文件, 节点1:

[2018-07-26 11:21:41,151][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480863] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[2018-07-26 11:21:45,652][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480875] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
:q
[2018-07-26 11:21:50,153][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480887] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

节点2:

[2018-07-26 11:21:41,151][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480863] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[2018-07-26 11:21:45,652][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480875] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
:q
[2018-07-26 11:21:50,153][WARN ][discovery.zen.ping.unicast] [node-1] failed to send ping to [{#zen_unicast_2#}{**.**.81.194}{**.**.81.194:9300}]

ReceiveTimeoutTransportException[[][**.**.81.194:9300][internal:discovery/zen/unicast] request_id [210480887] timed out after [3750ms]]

	at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

节点1,2提示无法ping通节点3, 登录到节点3上执行:

ps -ef|grep elastic

重启elasticsearch, 因为elasticsearch没有重启命令, 只能根据ps出的进程号杀掉进程, 然后执行:

./elasticsearch -d

中途可能会报错:

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

因为安全问题elasticsearch 不让用root用户直接运行，所以要创建新用户:

第一步：liunx创建新用户 adduser XXX 然后给创建的用户加密码 passwd XXX 输入两次密码。

第二步：切换刚才创建的用户 su XXX 然后执行elasticsearch 会显示Permission denied 权限不足。

第三步：给新建的XXX赋权限，chmod 777 * 这个不行，因为这个用户本身就没有权限，肯定自己不能给自己付权限。所以要用root用户登录付权限。

第四步：root给XXX赋权限，chown -R XXX /你的elasticsearch安装目录。

如果提示:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /server/pd/elasticsearch-2.3.3/logs/cc.log (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
	at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
	at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:243)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
log4j:ERROR Either File or DatePattern options are not set for appender [file].

表示没有写入日志的权限, 把日志权限赋给该用户即可: