1、Elastic Search安装
本次部署目录/data/elk下
cd /data/elk
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.1-linux-x86_64.tar.gz
解压到当前目录
tar -zxvf elasticsearch-8.4.1-linux-x86_64.tar.gz
编辑elasticsearch.yml文件
cd /data/elk/elasticsearch-8.4.1/config/
vi elasticsearch.yml
新增以下内容
network.host: 0.0.0.0 #可远程访问
node.name: node-base #节点名称 这个与下面一点一定要配,不然即使启动成功也会操作超时或发生master_not_discovered_exception
cluster.initial_master_nodes: ["node-base"] #发现当前节点名称
path.conf: /data/ELK/elasticsearch/conf
path.data: /data/ELK/elasticsearch/data
http.port: 9200 #端口号
http.cors.allow-origin: "*" #以下皆是跨域配置
http.cors.enabled: true
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true
ElasticSearch启动不能用root账号,需要新建账户
创建新用户及授权
groupadd elsearch #增加用户组
useradd elsearch -g elsearch -p elsearch #新增用户并设置用户组
cd /data/elk
chown -R elsearch:elsearch elasticsearch-7.5.1 #修改文件属所用户
su elsearch
cd /data/elk/elasticsearch-8.4.1/bin/
sh elasticsearch > /data/elk/el.log &
启动过程中出现如下错误
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
解决1:将当前用户的软硬限制调大
切换到root用户
su
vi /etc/security/limits.conf
新增以下配置
* soft nofile 65535
* hard nofile 65535
#也可以用@加账号
@elsearch soft nofile 65535
@elsearch hard nofile 65535
#执行
ulimit -n 65535
ulimit -n
#返回 65535
ulimit -H -n 65535
ulimit -H -n
#返回 65535
解决2:调大elasticsearch用户拥有的内存权限
切换到root用户
su
#临时修改
sysctl -w vm.max_map_count=262144
sysctl -a|grep vm.max_map_count #查看修改结果
#永久修改
vi /etc/sysctl.conf
#增加以下内容
vm.max_map_count=262144
#保存并执行
sysctl -p
解决3:没有配置node节点名称和主节点
修改配置文件
cd /data/elk/elasticsearch-8.4.1/config/
vi elasticsearch.yml
#修改以下配置
node.name: node-1
cluster.initial_master_nodes: ["node-1"]
#重新启动
sh elasticsearch &
出现以下画面启动成功
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.
ℹ️ Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
phtF6nPnroGhPCC_n9s9
ℹ️ HTTP CA certificate SHA-256 fingerprint:
7c6ff0a7ab717859c2d550fc3c06d0128b67791750e08ca70076cc75bd1dafc4
ℹ️ Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxOTIuMTY4LjEwMS4xMDI6OTIwMCJdLCJmZ3IiOiI3YzZmZjBhN2FiNzE3ODU5YzJkNTUwZmMzYzA2ZDAxMjhiNjc3OTE3NTBlMDhjYTcwMDc2Y2M3NWJkMWRhZmM0Iiwia2V5Ijoid01ySS1ZSUJzaEdRSzlZalE2a1g6YmxFMW1TeXpRZFNxcUZHYWUzcHJldyJ9
ℹ️ Configure other nodes to join this cluster:
• Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
eyJ2ZXIiOiI4LjQuMSIsImFkciI6WyIxOTIuMTY4LjEwMS4xMDI6OTIwMCJdLCJmZ3IiOiI3YzZmZjBhN2FiNzE3ODU5YzJkNTUwZmMzYzA2ZDAxMjhiNjc3OTE3NTBlMDhjYTcwMDc2Y2M3NWJkMWRhZmM0Iiwia2V5Ijoid3NySS1ZSUJzaEdRSzlZalE2a2k6Z2FOTzlEZVdSVnE1SGYtSWxLdFRyQSJ9
If you're running in Docker, copy the enrollment token and run:
修改配置文件elasticsearch.yml
ingest.geoip.downloader.enabled: false #新增
xpack.security.enabled: true #修改为false
xpack.security.http.ssl:
enabled: true #修改为false
keystore.path: certs/http.p12
重新启动
访问http://ip:9200
如果不能访问请查看防火墙是否开启9200端口
firewall-cmd --zone=public --add-port=9200/tcp --permanent
自启配置
进入进入到 cd /etc/init.d 目录
cd /etc/init.d 【进入到目录】
vi elsearch 【创建es系统启动服务文件】
添加以下内容,ES_HOME为es安装目录
su es为运行用户
#!/bin/bash
#chkconfig: 345 63 37
#description: elasticsearch
#processname: elasticsearch-8.4.1
export ES_HOME=/usr/local/elasticsearch-8.4.1
case $1 in
start)
su elsearch<<!
cd $ES_HOME
./bin/elasticsearch -d -p pid
exit
!
echo "elasticsearch is started"
;;
stop)
pid=`cat $ES_HOME/pid`
kill -9 $pid
echo "elasticsearch is stopped"
;;
restart)
pid=`cat $ES_HOME/pid`
kill -9 $pid
echo "elasticsearch is stopped"
sleep 1
su es<<!
cd $ES_HOME
./bin/elasticsearch -d -p pid
exit
!
echo "elasticsearch is started"
;;
*)
echo "start|stop|restart"
;;
esac
exit 0
修改权限
chmod 777 elsearch
添加和删除服务并设置启动方式
chkconfig --add elsearch 【添加系统服务】
chkconfig --del elsearch 【删除系统服务】
关闭和启动服务
service elsearch start 【启动】
service elsearch stop 【停止】
service elsearch restart 【重启】
设置服务是否开机启动
chkconfig elsearch on 【开启】
chkconfig elsearch off 【关闭】
2、Kibana安装
切换到root用户,下载安装包
su
cd /data/elk
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.5.1-linux-x86_64.tar.gz
解压到当前目录
tar -zxvf kibana-7.5.1-linux-x86_64.tar.gz
编辑kibana.yml文件
cd /data/elk/kibana-7.5.1-linux-x86_64/config/
vi kibana.yml
新增以下内容
server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana"
elasticsearch.url: "http://localhost:9200"
kibana.index: ".kibana"
保存并启动kibana
cd /data/elk/kibana-7.5.1-linux-x86_64/bin
#root不能直接启动kibana所以后面加--allow-root
sh kibana --allow-root > /data/elk/kibana.log &
启动出现错误
FATAL Error: [elasticsearch.url]: definition for this key is missing
修改kibana.yml文件
elasticsearch.url: "http://localhost:9200"
#修改成下面
elasticsearch.hosts: ["http://localhost:9200"]
访问http://ip:5601
如果不能访问请查看防火墙是否开启5601端口
3、Logstash安装
切换到root用户,下载安装包
su
cd /data/elk
wget https://artifacts.elastic.co/downloads/kibana/logstash-7.5.1.tar.gz
解压到当前目录
tar -zxvf logstash-7.5.1.tar.gz
新增logstash.conf文件
cd /data/elk/logstash-7.5.1/bin/
vi logstash.conf
新增以下内容
input {
tcp {
port => 5044
codec => json_lines
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
保存并启动logstash
cd /data/elk/logstash-7.5.1/bin/
sh logstash -f logstash.conf &
查看日志,无报错信息,默认启动成功。
4、配置项目
添加logstash依赖
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>6.3</version>
</dependency>
配置logback-spring.xml文件
加spring后缀是能使用标签
多环境配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProfile name="exp">
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>192.168.5.19:5044</destination>
<queueSize>1048576</queueSize>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<timeZone>UTC</timeZone>
</timestamp>
<pattern>
<pattern>
{
"severity":"%level",
"service": "wbx-gateway",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message->%ex{full}"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
</springProfile>
<springProfile name="test">
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>192.168.5.19:5044</destination>
<queueSize>1048576</queueSize>
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<timeZone>UTC</timeZone>
</timestamp>
<pattern>
<pattern>
{
"severity":"%level",
"service": "wbx-gateway",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message->%ex{full}"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
</springProfile>
<property name="pattern" value="%d{yyyy-MM-dd:HH:mm:ss.SSS} [%thread] %-5level %msg%n"/>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- 字符串System.out(默认)或者System.err -->
<target>System.out</target>
<!-- 对记录事件进行格式化 -->
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>${pattern}</pattern>
</encoder>
</appender>
<springProfile name="dev">
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</springProfile>
<springProfile name="exp">
<root level="info">
<appender-ref ref="LOGSTASH" />
</root>
</springProfile>
<springProfile name="test">
<root level="info">
<appender-ref ref="LOGSTASH" />
</root>
</springProfile>
</configuration>