elk(都是6.2.4重点-版本2-收集nginx日志并分析绘图（单点es,redis缓存）-无filebeat

elk(都是6.2.4重点-版本2-收集nginx日志并分析绘图（单点es,redis缓存）-无filebeat

如果对运维课程感兴趣，可以在b站上、A站或csdn上搜索我的账号： 运维实战课程，可以关注我，学习更多免费的运维实战技术视频

1.搭建elk6.2.4环境（cent7.4），logstash收集nginx日志写入es（用redis解耦操作）

规划： 192.168.171.128： es

       192.168.171.129： logstash1 和nginx (收集nginx日志) logstash1将日志数据写入redis

       192.168.171.130: nginx+kibana（nginx使得kibana能用用户和密码认证登录）

       192.168.171.131: redis 和logstash2         logstash2将日志数据从redis读出，写入es中

1)在192.168.171.128上:安装elasticsearch:

a)环境准备

[root@bogon ~]# vim /etc/security/limits.conf

* soft nofile 65536     #注意必须是65536以上，65535都不行

* hard nofile 65536

* soft nproc 4096

* hard nproc 4096

[root@bogon ~]# vim /etc/security/limits.d/20-nproc.conf

*          soft    nproc     4096        #至少是4096，若小于则需修改，也可大于4096

[root@bogon ~]# vim /etc/sysctl.conf

vm.max_map_count=262144      #添加，至少是262144

[root@bogon ~]# sysctl -p

[root@bogon ~]# reboot

[root@bogon ~]# systemctl stop firewalld.service

b)安装jdk1.8：

[root@bogon ~]# mkdir /usr/local/java

[root@bogon ~]# rz

上传jdk1.8包

[root@bogon ~]# ls

jdk-8u144-linux-x64.tar.gz

[root@bogon ~]# tar -zxf jdk-8u144-linux-x64.tar.gz

[root@bogon ~]# ls

jdk1.8.0_144  jdk-8u144-linux-x64.tar.gz

[root@bogon ~]# mv jdk1.8.0_144/ /usr/local/java/

[root@bogon ~]# ls /usr/local/java/

jdk1.8.0_144

[root@bogon ~]# vim /etc/profile

#最后面添加：

JAVA_HOME=/usr/local/java/jdk1.8.0_144

CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

PATH=$JAVA_HOME/bin:$PATH

export JAVA_HOME CLASSPATH PATH

[root@bogon ~]# source  /etc/profile

[root@bogon ~]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

[root@bogon ~]# rz

上传elasticsearch包，并安装elasticsearch，（es启动需要普通用户，其他服务启动用root）

[root@bogon ~]# ls

elasticsearch-6.2.4.tar.gz  jdk-8u144-linux-x64.tar.gz

[root@bogon ~]# tar -zxf elasticsearch-6.2.4.tar.gz

[root@bogon ~]# ls

elasticsearch-6.2.4  elasticsearch-6.2.4.tar.gz  jdk-8u144-linux-x64.tar.gz

[root@bogon ~]# mv elasticsearch-6.2.4 /opt/

[root@bogon ~]# ls /opt/elasticsearch-6.2.4/

bin  config  lib  LICENSE.txt  logs  modules  NOTICE.txt  plugins  README.textile

[root@bogon ~]# groupadd eszu

[root@bogon ~]# useradd esyonghu -g eszu -p 123456

[root@bogon ~]# chown -R esyonghu:eszu /opt/elasticsearch-6.2.4/

[root@bogon ~]# su - esyonghu

[esyonghu@bogon ~]$ vim /opt/elasticsearch-6.2.4/config/elasticsearch.yml

network.host: 192.168.171.128     #修改,本机ip,或者0.0.0.0,监听本机所有ip

http.port: 9200                   #释放

[esyonghu@bogon ~]$ cd /opt/elasticsearch-6.2.4/bin/

[esyonghu@bogon bin]$ ls elasticsearch

elasticsearch

[esyonghu@bogon bin]$ ./elasticsearch -d     #后台启动es，或者先./ elasticsearch前台启动查看下是否能启动再后台启动

[esyonghu@bogon bin]$ tailf /opt/elasticsearch-6.2.4/logs/elasticsearch.log  #查看启动日志，有时需要等会才能形成日志

………

[2019-10-12T21:27:15,635][INFO ][o.e.c.s.ClusterApplierService] [koccs5f] new_master {koccs5f}{koccs5fwSre801IprBSrzw}{A-z15PqtRByWA494KhVH_A}{192.168.171.128}{192.168.171.128:9300}, reason: apply cluster state (from master [master {koccs5f}{koccs5fwSre801IprBSrzw}{A-z15PqtRByWA494KhVH_A}{192.168.171.128}{192.168.171.128:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

[2019-10-12T21:27:15,743][INFO ][o.e.h.n.Netty4HttpServerTransport] [koccs5f] publish_address {192.168.171.128:9200}, bound_addresses {192.168.171.128:9200}

[2019-10-12T21:27:15,744][INFO ][o.e.n.Node               ] [koccs5f] started

[2019-10-12T21:27:15,746][INFO ][o.e.g.GatewayService     ] [koccs5f] recovered [0] indices into cluster_state

[esyonghu@bogon bin]$ netstat -anput |grep 9200

有9200端口

[esyonghu@bogon bin]$ curl http://192.168.171.128:9200

能访问es默认页面

客户端浏览器外网访问，如下：http://192.168.171.128:9200/ 

c)elasticsearch安装head插件：是es的一个客户端管理工具（es5.0版本后，head插件独立安装）

安装nodejs环境：  (head插件会用到grunt,而grunt需要npm包管理器，所以需要安装nodejs环境)

另开一个窗口,用root用户，也可用其他用户

[root@bogon ~]# rz

上传nodejs包

[root@bogon ~]# ls

elasticsearch-6.2.4.tar.gz  jdk-8u144-linux-x64.tar.gz  node-v9.6.1-linux-x64.tar.xz

[root@bogon ~]# xz -d node-v9.6.1-linux-x64.tar.xz

[root@bogon ~]# ls

elasticsearch-6.2.4.tar.gz  jdk-8u144-linux-x64.tar.gz  node-v9.6.1-linux-x64.tar

[root@bogon ~]# tar -xf node-v9.6.1-linux-x64.tar

[root@bogon ~]# ls

elasticsearch-6.2.4.tar.gz  jdk-8u144-linux-x64.tar.gz  node-v9.6.1-linux-x64  node-v9.6.1-linux-x64.tar

[root@bogon ~]# mv node-v9.6.1-linux-x64 /opt/

[root@bogon ~]# ls /opt/node-v9.6.1-linux-x64/

bin  CHANGELOG.md  include  lib  LICENSE  README.md  share

[root@bogon ~]# export PATH=$PATH:/opt/node-v9.6.1-linux-x64/bin

[root@bogon ~]# echo "export PATH=$PATH:/opt/node-v9.6.1-linux-x64/bin" >> /etc/profile

[root@bogon ~]# tail -1 /etc/profile

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/node-v9.6.1-linux-x64/bin:/opt/node-v9.6.1-linux-x64/bin

[root@bogon ~]# source  /etc/profile

[root@bogon ~]# node -v

v9.6.1

[root@bogon ~]# npm -v

5.6.0

[root@bogon ~]# npm install -g grunt-cli     #安装head插件依赖工具

[root@bogon ~]# grunt --version

grunt-cli v1.3.2

从github删下载head插件，也可先下载后，保留存起来

[root@bogon ~]# yum -y install git

[root@bogon ~]# git clone git://github.com/mobz/elasticsearch-head.git

[root@bogon ~]# ls

elasticsearch-6.2.4.tar.gz  elasticsearch-head  jdk-8u144-linux-x64.tar.gz  node-v9.6.1-linux-x64.tar

[root@bogon ~]# mv elasticsearch-head/ /opt      #移到/opt/elasticsearch-6.2.4/plugins/试验不行，新版可能不让在该目录下安装插件

[root@bogon ~]# ls /opt/

elasticsearch-6.2.4  elasticsearch-head  node-v9.6.1-linux-x64

[root@bogon ~]# cd /opt/elasticsearch-head/

[root@bogon elasticsearch-head]# ls

Dockerfile                          Gruntfile.js       LICENCE                       proxy           src

Dockerfile-alpine                   grunt_fileSets.js  package.json                  README.textile  test

elasticsearch-head.sublime-project  index.html         plugin-descriptor.properties  _site

更换npm源，安装cnpm命令，使用cnpm命令安装head插件

[root@bogon elasticsearch-head]# npm install -g cnpm --registry=https://registry.npm.taobao.org

[root@bogon elasticsearch-head]# cnpm install   #使用cnpm安装

……下面有些报错，但是没影响

Error: post install error, please remove node_modules before retry!

Run "sh -c node install.js" error, exit code 1

    at ChildProcess.proc.on.code (/opt/node-v9.6.1-linux-x64/lib/node_modules/cnpm/node_modules/runscript/index.js:74:21)

    at ChildProcess.emit (events.js:127:13)

……

[root@bogon elasticsearch-head]# vim Gruntfile.js

……

                connect: {

                        server: {

                                options: {

                                        hostname: '*',  #添加

                                        port: 9100,

                                        base: '.',

                                        keepalive: true

                                }

                        }

                }

[root@bogon elasticsearch-head]# vim _site/app.js

……

this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.171.128:9200"; #修改成本机ip

配置es允许跨域访问：（修改es的配置文件）

[root@bogon elasticsearch-head]# vim /opt/elasticsearch-6.2.4/config/elasticsearch.yml

#最后面添加两个下面的

http.cors.enabled: true        #添加

http.cors.allow-origin: '*'    #添加

wq

d)重新启动es服务: （可以在另一个普通用户的窗口）

[esyonghu@bogon bin]$ ps -ef |grep elasticsearch

查看进程号：1311

[esyonghu@bogon bin]$ kill -9 1311

[esyonghu@bogon bin]$ pwd

/opt/elasticsearch-6.2.4/bin

[esyonghu@bogon bin]$ ./elasticsearch -d

[esyonghu@bogon bin]$ tailf /opt/elasticsearch-6.2.4/logs/elasticsearch.log

[esyonghu@bogon bin]$ netstat -anput |grep 9200

有端口

e)启动es-head插件：（另一个终端，可以用root用户启动）

[root@bogon elasticsearch-head]# cd /opt/elasticsearch-head/node_modules/grunt/bin/

[root@bogon bin]# ls

grunt

[root@bogon bin]# ./grunt server &         #启动head插件服务

………

Running "connect:server" (connect) task