centos7 安装elk日志分析系统

https://blog.youkuaiyun.com/guyan0319/article/details/78749639

                    版权声明：转载请注明出处 http://blog.youkuaiyun.com/guyan0319                        https://blog.youkuaiyun.com/guyan0319/article/details/78749639                    </div>
                                                <link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/ck_htmledit_views-3019150162.css">
                                    <div id="content_views" class="markdown_views">
                <!-- flowchart 箭头图标 勿删 -->
                <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                    <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                </svg>
                                        <p>架构图 <br>

Elasticsearch：搜索，提供分布式全文搜索引擎；

Logstash: 日志收集，管理，存储；

Kibana ：日志的过滤web 展示；

Filebeat：监控日志文件、转发，其已取代 logstash forwarder；

一、准备工作
设置 yum源，采用官网提供的源
https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
下载并安装公共签名密钥：

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

 
 

 
 
1
2

创建yum的repo文件

vim    /etc/yum.repos.d/elasticsearch.repo
 
 

 
 
1

内容如下

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
 
 

 
 
1
2
3
4
5
6
7
8

二、elasticsearch安装
elasticsearch依赖Java开发环境支持，先安装JDK。

yum -y install java-1.8.0-openjdk
 
 

 
 
1

查看java安装情况

安装Elasticsearch

yum -y install elasticsearch
systemctl start elasticsearch
 
 

 
 
1
2

ElasticSearch默认的对外服务的HTTP端口是9200，节点间交互的TCP端口是9300。

ss -tlnp |grep -E '9200|9300'
 
 

 
 
1

测试服务

curl -X GET http://localhost:9200
 
 

 
 
1

三、安装Logstash

yum -y install logstash
systemctl start logstash
 
 

 
 
1
2

四、安装Kibana

yum -y install kibana
systemctl start kibana
 
 

 
 
1
2

五、浏览器http://localhost:5601

六、配置nginx 访问

vim /etc/nginx/conf.d/kibana.conf

server {
    listen       80;
    server_name  kb.com;
    access_log  /var/log/nginx/kibana.aniu.co.access.log;
    error_log   /var/log/nginx/kibana.aniu.co.access.log;
    #auth_basic "Restricted Access";
    #auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

 
 

 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

systemctl reload nginx
 
 

 
 
1

访问 http://kb.com

七、安装filebeat

yum  -y install  filebeat 
systemctl start filebeat 
systemctl enable filebeat
 
 

 
 
1
2
3

配置 Filebeat
vim /etc/filebeat/filebeat.yml ##配置filebeat

#============= Filebeat prospectors ===============
filebeat.prospectors:
- input_type: log
  enabled: true ＃更改为true以启用此prospectors配置。
  paths:
    #- /var/log/*.log
    - /var/log/messages
#==================== Outputs =====================
#------------- Elasticsearch output ---------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
#---------------- Logstash output -----------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]
 
 

 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

注意：要注释以下两行

output.elasticsearch和output.logstash只能同时开启一个
并且设置
enabled: true ＃更改为true以启用此prospectors配置。

systemctl restart filebeat
 
 

 
 
1

八、配置logstash
创建配置文件
vim /etc/logstash/conf.d/01-logstash-initial.conf

input {
  beats {
    port => 5044
    type => "logs"
  }
}

filter {
  #if [type] == "sy" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    geoip {
      source => "clientip"
    }
    syslog_pri {}
    date {
      match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
 # }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
 
 

 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

查看端口
ss -tlnp|grep -E ‘5044|9600’

验证logstash配置文件

/usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/01-logstash-initial.conf --config.test_and_exit
 
 

 
 
1

显示Configuration OK 证明配置成功
如果报错：WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using
解决办法：

cd /usr/share/logstash
ln -s /etc/logstash ./config
 
 

 
 
1
2

九、配置kibana
添加索引

查看状态图

参考资料：
https://www.elastic.co/guide/index.html
http://www.cnblogs.com/hanyifeng/p/5509985.html
http://blog.51cto.com/wangzhijian/1878636#comment