filebeat采集日志到ELK

1、安装

官方下载最新版，上传服务器解压

tar -zxvf filebeat-7.17.4-linux-x86_64.tar.gz

2、打印到控制台调试

1. 配置stdout.yml

# 输入
filebeat.inputs:
  # 标准输入
- type: stdin
  enabled: true
# 输出
# 输出到控制台
output.console:
  pretty: true
  enable: true

2、启动命令

./filebeat -e -c stdout.yml

3、采集到Elasticsearch

1）配置filebeat.yml

filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/access*.log
# filebeat收集java多行日志
# multiline.pattern: ^\[ #因为日志中得开头都是日期各式的，用这个去匹配
# multiline.negate: true  #开启多行模式
# multiline.match: after
  tags: ["access"]
  encoding: utf-8
  prospector.scanner.exclude_files: ['.gz$']
  fields_under_root: true
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: message
##系统日志
- type: filestream
  id: sys-log
  enabled: false
  paths:
    - /var/log/syslog*.log
  tags: ["sys-log"]
  encoding: utf-8
  prospector.scanner.exclude_files: ['.gz$']
#  fields_under_root: true

#json.keys_under_root: true        # 开启json格式      
#json.overwrite_keys: true
setup.ilm.enabled: false
setup.template.enabled: false # 不用模块版，只用我这里配置的
setup.template.settings:
  index.number_of_shards: 3

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "${filebeat_user}"
  password: "${filebeat_passwd}"
  indices:
    - index: "filebeat-access-%{+yyyy.MM.dd}"
      when.contains: #通过when进行判断,当标签是某个的时候就创建相应的索引 
         tags: "access"
    - index: "filebeat-sys-log-%{+yyyy.MM.dd}"
      when.contains:
         tags: "sys-log"

# setup.template.enabled: false
# setup.template.name: "filebeat"
# setup.template.pattern: "filebeat-*"
# setup.template.overwrite: true
#nginx 配置         
# setup.template.enabled: false
# setup.template.name: "nginx"
# setup.template.pattern: "nginx-*"
# setup.template.overwrite: true

4、开机启动配置

1. 正常启动

nohup ./filebeat -e -c xxxx.yml >> filebeat.log &

or

nohup ./filebeat -c ./filebeat.yml -e > /dev/null 2>&1 &

2. 开机自启动配置（CentOS 7.X）

• 新建服务

vi /usr/lib/systemd/system/filebeat.service

• 编辑文档

[Unit]
Description=filebeat
Wants=network-online.target
After=network-online.target
[Service]
User=root
ExecStart=/opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml
Restart=always  #设置为掉线自动重启，进程强制杀掉后会自动重新启动
[Install]
WantedBy=multi-user.target

• 启动服务

systemctl start filebeat.service
systemctl enable filebeat.service

systemctl daemon-reload  　　　　　　　　   #加载配置
systemctl enable filebeat             　　#设置开机自启动
systemctl disable filebeat            　　#停止开机自启动

systemctl start filebeat              　　#启动filebeat服务
systemctl restart filebeat　          　　#重新启动服务

systemctl status filebeat             　　#查看服务当前状态
systemctl list-units --type=service      #查看所有已启动的服务

5、keystore使用

用于filebeat.yml中动态获取参数

#签名文件
filebeat keystore create
#新增密码
filebeat keystore add ES_PWD
#更新签名文件
filebeat keystore add ES_PWD --force
#签名文件 列表
filebeat keystore list
#移除签名文件
filebeat keystore remove ES_PWD

6、遇到的坑

检测到了log的变化，但kibana和ES上没有查到相应的索引文件。

ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://x.x.x.x:9200)): Connection marked as failed because the onConnect callback failed: error loading template: failure while checking if template exists: 405 Method Not Allowed:

[elasticsearch] elasticsearch/client.go:414     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.June, 8, 18, 8, 9, 18041600, time.Local), Meta:null,Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=404): {"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([.security,.security-6,.monitoring-*,.watch*,.triggered_watches,.quota]) doesn't match","index_uuid":"_na_","index":"logstash-ld456-access-2022.06.08"}, dropping event!

解决办法是开启自动创建index的配置，或者根据报错信息手动创建一个索引

PUT /_cluster/settings
{
    "persistent" : {
        "action": {
          "auto_create_index": "true"
        }
    }
}

7、仅保留message消息，且去掉其他字段

方式一

processors:
  - decode_json_fields:
        fields: ["message"]
        target: ""
  - drop_fields:
        fields: ["ecs","cloud","host","agent"]
        ignore_missing: true

方式二

filebeat.inputs:
- type: filestream
  ...
  parsers:
    - ndjson:
        target: ""
        message_key: msg
    - multiline:
        type: counter
        lines_count: 3

官方参考文档：https://www.elastic.co/guide/en/beats/filebeat/7.17/elasticsearch-output.html