较详细的filebeat+es+kibana+sentinl，日志采集+mail、钉钉报警

本文链接：https://blog.youkuaiyun.com/qq_39506978/article/details/102829487

1.总体架构图

2.日志截图，filebeat需要采集json日志行。中文变量是为了可读性。JSON.toJsonstring(obj,true)进行pretty打印JSON

3.filebeat配置，多使用 ./filebeat -e -c filebeat.yml -d '*' 查看采集状况，filebeat自带一些常用模块如mysql慢查询，错误日志。进入filebeat/modules.d/ 去掉要使用的模块.disabled后缀然后修改配置。 ./filebeat setup -e即可将默认模板写入kibana


filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /home/work/logs/*Log/all_level/all_level.log

  #tags: jwanj
  include_lines: ['^ +'] #只采集空格开头的行
  #exclude_lines: ['ERROR','INFO','DEBUG'] #排除行
  multiline.pattern: "^ +{"  #匹配空格{开头的json行
  multiline.negate: true
  multiline.match: after
  scan_frequency: 10s

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
#============================== Dashboards =====================================
setup.dashboards.enabled: false
#============================== Kibana =====================================
setup.kibana:
  host: "4x.xxx.xx.18:8101"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]
  pipelines:
    - pipeline: logaop     #指定es ingest节点的日志清洗管道
  username: "elastic"
  password: "6esDhSTr8lXXXXXXX"

4.es ingest 节点创建pipeline

grok抽取匹配的json串，并json格式化，geoip是es的一个插件 ./elasticsearch-plugin install ingest-geoip,不需要安装 GeoLite2-City.mmdb这个城市数据库。on_failure这个配置非常必要，pipeline处理异常的信息会存入failed-{{ _index }}这个索引，如果没有配置这个会导致filebeat不断报错而不能继续处理后面的日志。

PUT /_ingest/pipeline/logaop
{
	"description" : "logaop",
    "processors" : [
        {
           <!-- "grok": {
              "field": "message",
              "patterns": [".*logaop=>%{GREEDYDATA:jsonContent}"]
            },不需要了 -->
            "json":{
              "field": "message",
              "target_field": "content", 
              "add_to_root": false
            },
            "geoip":{
              "field":"content.请求IP",
              "target_field":"geoip"
            },
           
            "remove": {
              "field": ["message","beat","prospector","geoip.continent_name","geoip.country_iso_code","geoip.region_name"]
            }
           
            
        }
    ],
    "on_failure": [
		{
			"set": {
				"field": "_index",
				"value": "failed-{{ _index }}"
			}
		}
	]
}

5.监控大盘，步骤4处理出来的geoip.location并不是geo_point类型的，暂未解决，导致请求热力图无法画出

6.报警。最初打算使用elastAlert的，安装特别坑，就换sentinel了,这个两个报警方案都暂时不支持高版本7.xES，sentinl github地址=>https://github.com/sirensolutions/sentinl/releases/ 。安装方法类似kibana安装xpack插件,推荐在线安装./kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.2.4/sentinl.zip。注意版本一致，这里所有的组件都是08年的6.2.4

要获取payload的列表信息时需要使用transform，里面的script是js，如下即是将循环命中的文档将_source.content字段转json后存入数组，发送邮件就可以{{payload.newlist}}获取

"script": "payload.newlist=[];payload.hits.hits.forEach(function(bucket){ payload.newlist.push(JSON.stringify(bucket._source.content))})"

QQ邮件发送服务器stmp.qq.com的25端口可能被阿里云服务器封了，可以ping通但Telnet测试端口是不通的，当然也可以解决但比较麻烦，推荐考虑钉钉报警或微信，钉钉需要去pc端钉钉获取一个机器人webhook。配置时body配置参考钉钉文档https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq。

{
  "actions": {
    "email_html_alarm_2b622930-10e5-11ea-8156-a34196b5558f": {
      "name": "email html alarm",
      "throttle_period": "1m",
      "email_html": {
        "to": "523083921@qq.com;",
        "from": "523083921@qq.com",
        "stateless": false,
        "subject": "{{watcher.title}},异常数{{payload.hits.total}}",
        "priority": "medium",
        "html": "<p>Hi {{watcher.username}},</p>\n<p>{{watcher.title}},异常数{{payload.hits.total}},详情如下(应用---接口---耗时毫秒数)：</p> <p>{{payload.newlist}}</p>\n<p>Click <a target=\"_blank\" href=\"{{watcher.dashboard_link}}\">点我</a> 查看监控面板.</p>\n<div style=\"color:grey;\">\n  <hr />\n  <p>报警触发条件:</p>\n  <ul><li>{{watcher.condition.script.script}}</li></ul>\n</div>"
      }
    },
    "Webhook_93b7f699-8bb4-45c3-96b4-48e1fddf2f19": {
      "name": "Webhook1",
      "throttle_period": "1m",
      "webhook": {
        "priority": "high",
        "stateless": false,
        "method": "POST",
        "host": "oapi.dingtalk.com",
        "port": "443",
        "path": "/robot/send?access_token=XXXXXXX",
        "body": "{\"msgtype\": \"markdown\",\"at\": {\"isAtAll\": \"True\"},\"markdown\": {\"title\": \"{{watcher.title}},异常数{{payload.hits.total}}\",\"text\": \"**{{watcher.title}},异常数{{payload.hits.total}}。** \n\n #### 详情如下(应用---接口---耗时毫秒数)：\n > {{payload.newlist}}\n > ##### [查看dashbord](http://47.108.28.18:8101) \n \"}}",
        "params": {
          "watcher": "{{watcher.title}}",
          "payload_count": "{{payload.hits.total}}"
        },
        "headers": {
          "Content-Type": "application/json"
        },
        "message": "测试报警",
        "use_https": true
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "index": [
          "filebeat-*"
        ],
        "body": {
          "size": 100,
          "aggs": {
            "2": {
              "date_histogram": {
                "field": "@timestamp",
                "interval": "30s",
                "time_zone": "Asia/Shanghai",
                "min_doc_count": 1
              },
              "aggs": {
                "3": {
                  "max": {
                    "field": "content.请求耗时"
                  }
                }
              }
            }
          },
          "version": true,
          "_source": {
            "excludes": []
          },
          "stored_fields": [
            "*"
          ],
          "script_fields": {},
          "docvalue_fields": [
            "@timestamp"
          ],
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m/m",
                      "lte": "now/m"
                    }
                  }
                },
                {
                  "range": {
                    "content.请求耗时": {
                      "gte": 1000
                    }
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "script": "payload.hits.total>0"
    }
  },
  "transform": {
    "script": {
      "script": "payload.newlist=[];payload.hits.hits.forEach(function(bucket){ payload.newlist.push(bucket._source.content.应用名+'-----'+bucket._source.content.请求URI+'-----'+bucket._source.content.请求耗时)+'    ' })"
    }
  },
  "trigger": {
    "schedule": {
      "later": "every 10 minutes"
    }
  },
  "disable": true,
  "report": false,
  "title": "【测试环境】10分钟一次，接口耗时超过1s报警",
  "save_payload": false,
  "spy": true,
  "impersonate": false,
  "dashboard_link": "http://47.108.28.18:8101/app/kibana#/dashboard/5867fa70-fafd-11e9-8f31-bf285ea1d362?_g=(refreshInterval:('$$hashKey':'object:1714',display:'10%20seconds',pause:!f,section:1,value:10000),time:(from:now-15m,mode:quick,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(spy:!n),gridData:(h:3,i:'1',w:6,x:0,y:0),id:da698230-faf5-11e9-8f31-bf285ea1d362,panelIndex:'1',type:visualization,version:'6.2.4'),(embeddableConfig:(spy:(mode:(name:setalarm))),gridData:(h:3,i:'2',w:6,x:6,y:0),id:'2e3f0960-faf7-11e9-8f31-bf285ea1d362',panelIndex:'2',type:visualization,version:'6.2.4'),(embeddableConfig:(mapCenter:!(32.39851580247402,108.19335937500001),mapZoom:4),gridData:(h:3,i:'3',w:6,x:6,y:3),id:b7ee8390-fb09-11e9-8f31-bf285ea1d362,panelIndex:'3',title:%E4%BC%81%E4%B8%9A%E5%9C%B0%E5%9D%80%E5%88%86%E6%AD%A5%E5%9B%BE,type:visualization,version:'6.2.4'),(gridData:(h:3,i:'4',w:6,x:0,y:6),id:'50115a30-fb0a-11e9-8f31-bf285ea1d362',panelIndex:'4',type:visualization,version:'6.2.4'),(gridData:(h:3,i:'5',w:6,x:0,y:3),id:'1af42580-fb0f-11e9-8f31-bf285ea1d362',panelIndex:'5',type:visualization,version:'6.2.4')),query:(language:lucene,query:''),timeRestore:!f,title:%E6%9C%BA%E4%B8%87%E5%AE%B6%E6%97%A5%E5%BF%97%E5%A4%A7%E7%9B%98,viewMode:edit)"
}

kibana.yml增加sentinl配置项，sentinl详细配置参考官网https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq。egrep -v "^$|#" kibana.yml 去#注释及空行查看配置如下,password是QQ邮箱-设置-POP3/SMTP的授权码

tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
server.port: 8101
sentinl:
 settings:
   email:
     active: true
     user: 523083921@qq.com
     password: knozhtzhtodscbca
     host: smtp.qq.com
     ssl: false
   report:
     active: true
server.host: "1xx.xx.xx.205"
elasticsearch.url: "http://localhost:9200"
elasticsearch.username: "elastic"
elasticsearch.password: "6esDhSTr8lINSgjaliQf"

效果图