ELK-elasticsearch-部署

一 准备工作

环境
系统 主机名
CentOS Linux release 7.6 linux-node3.example.com
CentOS Linux release 7.6 linux-node4.example.com
CentOS Linux release 7.6 linux-node5.example.com

工具准备
同步yum源
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
安装工具和包
yum install -y net-tools vim lrzsz tree screen lsof tcpdump wget ntpdate
时间同步
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo “*/5 * * * * ntpdate time1.aliyun.com &> /dev/null && hwclock -w” >> /var/spool/cron/root

vmware 2个虚拟机node3、node4，分别添加一块15G的硬盘，并且挂到/elk这个目录

[root@promote ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 15G 1.2G 14G 8% /
devtmpfs 475M 0 475M 0% /dev
tmpfs 487M 0 487M 0% /dev/shm
tmpfs 487M 7.6M 479M 2% /run
tmpfs 487M 0 487M 0% /sys/fs/cgroup
/dev/sda1 1014M 133M 882M 14% /boot
tmpfs 98M 0 98M 0% /run/user/0
/dev/sdb 15G 33M 15G 1% /elk

准备好安装包，3台都要
[root@linux-node3 soft]# ls
elasticsearch-5.6.5.rpm jdk-8u151-linux-x64.rpm

二 安装

[root@linux-node3 soft]# yum install *.rpm
[root@linux-node4 soft]# yum install *.rpm
[root@linux-node5 soft]# yum install *.rpm
安装看io的工具
[root@linux-node3 soft]# yum install iotop
[root@linux-node4 ~]# vim /etc/elasticsearch/elasticsearch.yml —主配置文件
[root@linux-node4 ~]# vim /etc/elasticsearch/jvm.options ----调优的配置文件

[root@linux-node4 ~]# vim /etc/elasticsearch/elasticsearch.ym
cluster.name: elk-cluster1 —17行
node.name: elk-node2 ----23行
path.data: /elk/data ----33行
path.logs: /elk/logs -----37行
network.host: 10.0.0.18 ----55 行
http.port: 9200 —59 行
discovery.zen.ping.unicast.hosts: [“10.0.0.17”, “10.0.0.18”] ----68 行

检验下配置文件

[root@linux-node4 ~]# grep "^[a-Z]" /etc/elasticsearch/elasticsearch.yml 

cluster.name: elk-cluster1
node.name: elk-node2
path.data: /elk/data
path.logs: /elk/logs
network.host: 10.0.0.18
http.port: 9200
discovery.zen.ping.unicast.hosts: [“10.0.0.17”, “10.0.0.18”]

[root@linux-node4 ~]# systemctl start elasticsearch
[root@linux-node4 ~]# chown elasticsearch.elasticsearch /elk/
[root@linux-node4 ~]# systemctl start elasticsearch

看下是否成功

通过网页可以测试下

配置文件拷到node3上
[root@linux-node4 elk]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.17:/etc/elasticsearch
修改配置
[root@linux-node3 ~]# vim /etc/elasticsearch/elasticsearch.yml
node.name: elk-node1
network.host: 10.0.0.17
[root@linux-node3 ~]# chown elasticsearch.elasticsearch /elk/
[root@linux-node3 ~]# systemctl start elasticsearch

安装插件elasticsearch-head
[root@linux-node3 ~]# cd /usr/local/src/
[root@linux-node3 src]# ls --插件已装备好了
elasticsearch-head.tar.gz
PS:这里的tar包是老师装好的，如果是一次安装需要按照图下执行
例：
[root@linux-host1 ~]# cd /usr/local/src/
[root@linux-host1 src]#git clone git://github.com/mobz/elasticsearch-head.git
[root@linux-host1 src]# cd elasticsearch-head/
[root@linux-host1 elasticsearch-head]# yum install npm -y
[root@linux-host1 elasticsearch-head]# npm install grunt -save
[root@linux-host2 elasticsearch-head]# ll node_modules/grunt #确认生成文件
[root@linux-host1 elasticsearch-head]# npm install #执行安装
[root@linux-host1 elasticsearch-head]# npm run start & #后台启动服务

这里我开始安装这个插件
[root@linux-node3 src]# tar xvf elasticsearch-head.tar.gz
[root@linux-node3 src]# cd elasticsearch-head
[root@linux-node3 elasticsearch-head]# ll node_modules/grunt

[root@linux-node3 elasticsearch-head]# yum install npm
[root@linux-node3 elasticsearch-head]# npm run start &
[1] 9732

elasticsearch-head@0.0.0 start /usr/local/src/elasticsearch-head
grunt server

Local Npm module “grunt-contrib-jasmine” not found. Is it installed?

Running “connect:server” (connect) task
Waiting forever…
Started connect web server on http://localhost:9100

检查下有没有成功

加在最后，开始http访问和授权
[root@linux-node3 ~]# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: “*”

[root@linux-node3 ~]# systemctl restart elasticsearch
[root@linux-node3 elasticsearch-head]# ss –lnt
看看9100 这个端口有没有起来，如果没有进入/usr/local/src/elasticsearch-head,执行npm run start &
[root@linux-node4 ~]# vim /etc/elasticsearch/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: “*”
[root@linux-node4 ~]# systemctl restart elasticsearch

三 内存优化

[root@linux-node3 ~]# vim /etc/elasticsearch/jvm.options

#Xms represents the initial size of total heap space
#Xmx represents the maximum size of total heap space
-Xms2g
-Xmx2g

锁定内存的配置
[root@linux-node4 ~]# vim /etc/elasticsearch/elasticsearch.yml
#Lock the memory on startup:
bootstrap.memory_lock: true

[root@linux-node3 ~]# vim /usr/lib/systemd/system/elasticsearch.service --启动配置脚本
LimitMEMLOCK=infinity --添加
[root@linux-node3 ~]# systemctl restart elasticsearch
Warning: elasticsearch.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.
[root@linux-node3 ~]# systemctl daemon-reload
[root@linux-node3 ~]# systemctl restart elasticsearch

第二台也改好
[root@linux-node4 ~]# vim /etc/elasticsearch/jvm.options
-Xms256M
-Xmx256M

[root@linux-node4 ~]# vim /etc/elasticsearch/elasticsearch.yml
#Lock the memory on startup:
bootstrap.memory_lock: true

[root@linux-node4 ~]# vim /usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity
[root@linux-node4 ~]# systemctl daemon-reload
[root@linux-node4 ~]# systemctl restart elasticsearch

测试 简单学习下

四 模拟集群监控

[root@linux-node4 ~]# systemctl stop elasticsearch

通过shell命令获取集群状态
[root@linux-node3 ~]# curl –sXGET http://10.0.0.17:9200/_cluster/health?pretty=true
curl: (6) Could not resolve host: xn–sxget-xu3b; Unknown error
{
“cluster_name” : “elk-cluster1”,
“status” : “yellow”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 5,
“active_shards” : 5,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 5,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 50.0
}

#获取到的是一个json格式的返回值，那就可以通过python对其中的信息进行分析，例如对status进行分析，如果等于green(绿色)就是运行在正常，等于yellow(黄色)表示副本分片丢失，red(红色)表示主分片丢失
[root@linux-node3 ~]# vim elk-monitor.py

#!/usr/bin/env python
#coding:utf-8
#Author Zhang Jie

import subprocess
body = ""
false="false"
obj = subprocess.Popen(("curl -sXGET http://10.0.0.17:9200/_cluster/health?pretty=true"),shell=True, stdout=subprocess.PIPE)
data =  obj.stdout.read()
data1 = eval(data)
status = data1.get("status")
if status == "green":
    print "50"
else:
print "100"