ELK集群部署(一)之部署elasticsearch 7集群

elasticsearch部署架构
由于最近项目需要，我将介绍一下ELK集群部署的方法和维护办法。
请有需求的粉丝，查阅合集。

部署elasticsearch 7集群

ELK集群由3台虚拟机组成。

服务	IP	端口
ES集群	192.168.56.101 192.168.56.102 192.168.56.103	9200 9300
kafka zookeeper	192.168.56.103	9092 2181
logstash	192.168.56.102	5044
kibana	192.168.56.101	5601

虚拟机的需求cpu 要2核，不然后面logstash kafka 运行会报错

# 1. limits config file
echo "1. limits config file"
cat >> /etc/security/limits.conf <<EOF
* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536
EOF
echo "vm.max_map_count = 655360" >>/etc/sysctl.conf
sysctl -p

# 2. create user  & set password
echo "2. add user esuser"
useradd esuser
echo "esuser"| passwd --stdin esuser

# 3. stop firewall
echo "3. stop firewalld"
systemctl stop firewalld
systemctl disable firewalld.service

# 4. hosts file
echo "4. edit /etc/hosts file"
cat <<EOF > /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.5.101  node-1
192.168.5.102  node-2
192.168.5.103  node-3
EOF

# 5. download es 8.1.3 package
echo "5. download es 8.1.3"
cd  /usr/local/src/
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.1-linux-x86_64.tar.gz
tar zxf elasticsearch-7.7.1-linux-x86_64.tar.gz
chown -R esuser:esuser /usr/local/src/elasticsearch-7.7.1

# 6. create es directory
echo "6. create es directory"
mkdir -p /es/{data,log}
chown -Rf esuser:esuser /es

# 7. es config file
echo "7. edit es config file"
IP=$(ip a|grep "inet 192.168.56"|awk '{print $2 }'|awk -F'/' '{print $1}')
NODE=$(hostname)

cat >> /usr/local/src/elasticsearch-7.7.1/config/elasticsearch.yml <<EOF
cluster.name: es-cluster
node.name: ${NODE}
path.data: /es/data
path.logs: /es/logs
http.port: 9200
network.host: ${IP}
discovery.seed_hosts: ["192.168.56.101:9300","192.168.56.102:9300","192.168.56.103:9300"]
cluster.initial_master_nodes: ["node-1"]
EOF

通过上面的脚本部署ES集群，在3台机器上都要做。

启动ES

su - esuser
/usr/local/src/elasticsearch-7.7.1/bin/elasticsearch
可以看到，它跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。
使用ctrl+C停止。当然，也可以使用后台进程的方式启动ES：
su - esuser
nohup /usr/local/src/elasticsearch-7.7.1/bin/elasticsearch &

查看效果:

集群健康状态
http://192.168.56.101:9200/_cluster/health

结果中：关注颜色状态 green : 有完整的主版本和副本，一切正常； yellow : 数据完整，但是没有完整的副本，相当于警告； red : 数据不完整，部分主分片不可用，相当于故障；

查看节点状态
http://192.168.56.101:9200/_cat/nodes

192.168.5.102 75 93 6 0.00 0.10 0.14 cdfhilmrstw - node-2
192.168.5.101 60 86 4 0.00 0.07 0.13 cdfhilmrstw * node-1
192.168.5.103 47 92 4 0.00 0.09 0.13 cdfhilmrstw - node-3

可以看出 node-1 为master节点

查看节点健康状况
http://192.168.56.101:9200/_cat/health

1652441517 11:31:57 es-cluster1 green 3 3 4 2 0 0 0 0 - 100.0%

查看集群索引
curl -X GET http://192.168.56.101:9200/_cat/indices?pretty

green open .apm-agent-configuration          f-Qn9WU_RxG6jHUtzriXnw 1 1     0      0    416b    208b
green open .monitoring-logstash-7-2022.05.25 m96rF_z0Q2eqOJZtOPT82w 1 1 55379      0     6mb     3mb
green open .monitoring-logstash-7-2022.05.24 g9m4y2pSR-a5ryk3bicHDQ 1 1  8251      0   1.4mb 744.3kb
green open .kibana_1                         2mudPr7-QDeI1rY9Z6JYdg 1 1  1222      2 455.7kb 214.7kb
green open .monitoring-kibana-7-2022.05.20   dtmK36wtQbqWOpXPkcboGw 1 1  2442      0   1.3mb 689.7kb