Logstash-优快云博客

Logstash 是 Elastic Stack（ELK）的核心组件之一，主要用于日志的采集、处理和转发。你可以把它理解为“日志管道”的中间层，负责从多种来源收集数据、解析、过滤，然后输出到目标系统，如 Elasticsearch。

一、Logstash 安装

以 CentOS 7 为例

###添加 Elasticsearch GPG key 和 YUM 源
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

cat > /etc/yum.repos.d/logstash.repo << EOF
[logstash]
name=Elastic repository for Logstash
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF


yum install -y logstash

二、Logstash 基本结构

Logstash 配置文件分为三个主要部分：
input {
  # 数据来源
}
filter {
  # 数据过滤与处理
}
output {
  # 数据输出目的地
}

三、编写配置文件

例如从一个 JSON 文件收集日志，然后发送到 Elasticsearch：
input {
  file {
    path => "/var/log/demo.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"
  }
}

filter {
  mutate {
    add_field => { "project" => "demo-app" }
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.200.201:9200"]
    index => "demo-log-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}
参数说明
file 表示从文件读取日志
path 监听的日志文件路径，例如 /var/log/demo.log
start_position => "beginning" 从文件的开头开始读取日志（仅第一次有效）
sincedb_path => "/dev/null" 不记录“读取到哪儿了”，每次都从头读取（适合测试）
codec => "json" 表示日志每一行是 JSON 格式，会自动解析成字段

操作说明
mutate 一个常用的字段处理插件
add_field 给每条日志增加一个字段 "project": "demo-app"

参数	说明
`file`	表示从文件读取日志
`path`	监听的日志文件路径，例如 `/var/log/demo.log`
`start_position => "beginning"`	从文件的开头开始读取日志（仅第一次有效）
`sincedb_path => "/dev/null"`	不记录“读取到哪儿了”，每次都从头读取（适合测试）
`codec => "json"`	表示日志每一行是 JSON 格式，会自动解析成字段

操作	说明
`mutate`	一个常用的字段处理插件
`add_field`	给每条日志增加一个字段 `"project": "demo-app"`

四、启动 Logstash

1. 使用配置文件运行

logstash -f /etc/logstash/conf.d/log-to-es.conf

2. 后台运行（使用 systemd）

systemctl enable logstash
systemctl start logstash
systemctl status logstash

五、接收 Filebeat 日志转发到 ES

配置路径：/etc/logstash/conf.d/beats-to-es.conf

input {
  beats {
    port => 5044
  }
}

filter {
  if [fileset][module] == "nginx" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.200.201:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

六、logstash的过滤插件之geoip实战案例

#(1)logstash配置文件
[root@elk101  ~]# cat config/03-beats-geoip-es.conf 
input { 
  # 指定输入的类型是一个beats
  beats {
    # 指定监听的端口号
    port => 8888
  }
} 

filter {
  # 根据IP地址分析客户端的经纬度，国家，城市信息等。
  geoip {
     source => "clientip"
     remove_field => [ "agent","log","input","host","ecs","tags" ]
  }
}
output { 
  # 将数据在标准输出显示
  stdout {} 
  # 将数据写入ES集群
  elasticsearch {
    # 指定ES主机地址
    hosts => ["http://192.168.200.201:9200"]
    # 指定索引名称
    index => "xiaop-logstash"
  }
}

[root@elk101  ~]# logstash -rf config/03-beats-geoip-es.conf

filebeat采集数据到logstash

[root@elk103  filebeat-7.17.5-linux-x86_64]# cat config/18-nginx-to-logstash.yaml 
filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log*
  json.keys_under_root: true
  json.add_error_key: true

# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["192.168.200.201:8888"]

[root@elk103  filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/18-nginx-to-logstash.yaml

七、logstash解析nginx原生日志并分析IP地址实战

#(1)logstash配置文件编写
[root@elk101  ~]# cat config/04-beats-grok_geoip-es.conf 
input { 
  beats {
    port => 8888
  }
} 
filter {
   grok {
      match => { "message" => "%{HTTPD_COMBINEDLOG}" }
      remove_field => [ "agent","log","input","host","ecs","tags" ]
   }
  geoip {
     source => "clientip"
  }
}
output { 
 #  stdout {}   
  elasticsearch {
    hosts => ["http://192.168.200.201:9200"]
    index => "xiaop-logstash-nginx"
  }
}

[root@elk101  ~]# logstash -rf config/04-beats-grok_geoip-es.conf 
		
#(2)filbeat采集日志
[root@elk103  filebeat-7.17.5-linux-x86_64]# cat config/19-nginx-to-logstash.yaml
filebeat.inputs:
- type: log
  paths:
    - /tmp/xiaop/access.log
# 将数据输出到logstash中
output.logstash:
  # 指定logstash的主机和端口
  hosts: ["192.168.200.201:8888"]
  
[root@elk103  filebeat-7.17.5-linux-x86_64]# filebeat -e -c config/19-nginx-to-logstash.yaml

八、logstash的单分支和双分支

#[root@elk101  ~]# cat config/06-tcp-grok_custom_pattern-es.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }

  tcp {
    port => 9999
    type => "tcp"
  }

  http {
    type => "http"
  }
} 
filter {
  if [type] == "beats" {
      grok {
         match => { "message" => "%{HTTPD_COMBINEDLOG}" }
         remove_field => [ "agent","log","input","host","ecs","tags" ]
      }

      geoip {
         source => "clientip"
         add_field => {"custom-type" => "beats"}
      }
      
      date {
          match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
          timezone => "Asia/Shanghai"
          target => "xiaop-date"
      }
  }
if [type] == "tcp" {
     grok {
         # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
         patterns_dir => ["./xiaop-patterns"]
         # 基于指定字段进行匹配
         # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         add_field => {"custom-type" => "tcp"}
    }
  }else {
    mutate {
       add_field => { 
           "school" => "xiaop" 
           "class" => "2025"
           "custom-type" => "http"
       } 
    }
  }

}
output { 
 stdout {} #先输出到终端测试
 
 #  elasticsearch {
 #    hosts => ["http://192.168.200.201:9200"]
 #    index => "xiaop-logstash-nginx"
 #  }
}

九、logstash的多分支案例

[root@elk101  ~]# cat config/07-tcp-grok_custom_pattern_if-es.conf 
input { 
  beats {
    port => 8888
    type => "beats"
  }
  tcp {
    port => 9999
    type => "tcp"
  }
  http {
    type => "http"
  }
} 
filter {
  if [type] == "beats" {
      grok {
         match => { "message" => "%{HTTPD_COMBINEDLOG}" }
         remove_field => [ "agent","log","input","host","ecs","tags" ]
      }
      geoip {
         source => "clientip"
         add_field => {"custom-type" => "beats"}
      }      
      date {
          match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
          timezone => "Asia/Shanghai"
          target => "xiaop-date"
      }
  } else if [type] == "tcp" {
     grok {
         # 指定加载pattern匹配模式的目录，可以是相对路径，也可以是绝对路径
         patterns_dir => ["./xiaop-patterns"]
         # 基于指定字段进行匹配
         # match => { "message" => "%{TEACHER:teacher}edu%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         match => { "message" => "%{TEACHER:teacher}.{3}%{YEAR:year} 教室%{CLASSROOMNUMBER:classroom_number}"}
         add_field => {"custom-type" => "tcp"}
    }
  }else {
    mutate {
       add_field => { 
           "school" => "xiaop" 
           "class" => "2025"
           "custom-type" => "http"
       } 
    }
  }
}
output { 
 stdout {} 
 #  elasticsearch {     #这里先输出到终端
 #    hosts => ["http://192.168.200.201:9200"]
 #    index => "xiaop-logstash-nginx"
 #  }
}

####有待补充。。。。。