grok的简单使用

grok的简单使用

假设有日志记录如下

127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] "GET /favicon.ico HTTP/1.1" 404 3650 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0" "-"

处理目标：过滤出客户端IP字段

使用方法

1. 使用logstash自带的模式过滤

   编辑配置文件

   #输入
   input {
       generator {
           count => 1
           message => '127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] "GET /favicon.ico HTTP/1.1" 404 3650 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0" "-"'
       }   
   }
   #过滤
   filter {
       grok {
           match => { 
               "message" => "%{IPORHOST:clientip}"
           }   
       }   
   }
   
   #输出
   output {
       stdout {
           codec => rubydebug
       }   
   }
   
   

   输出结果

   {
             "host" => "localhost.localdomain",
       "@timestamp" => 2020-11-16T10:00:38.671Z,
         "@version" => "1",
         "clientip" => "127.0.0.1",
          "message" => "127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] \"GET /favicon.ico HTTP/1.1\" 404 3650 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0\" \"-\"",
         "sequence" => 0
   }
   

2. 自定义模式

   2.1、利用文件定义模式及调用
   压缩包方式安装的logstash自带的模式文件(相对)路径为
   vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns

   [yunwei@localhost patterns]$ pwd
   /data/software/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns
   [yunwei@localhost patterns]$ ls
   aws     bind  exim       grok-patterns  httpd  junos         maven        mcollective-patterns  nagios      rails  ruby
   bacula  bro   firewalls  haproxy        java   linux-syslog  mcollective  mongodb               postgresql  redis  squid
   

   不建议直接修改模式文件，建议新建文件

   vim selfpattern
   CLIENTIP \d{1,4}.\d{1,4}.\d{1,4}.\d{1,4}
   #格式： 名称 模式
   

   配置文件

   input {
       generator {
           count => 1
           message => '127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] "GET /favicon.ico HTTP/1.1" 404 3650 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0" "-"'
       }   
   }
   filter {
       grok {
   				#如果不在原来的路径，需要额外所在文件夹路径
   				#patterns_dir=>"path"
           match => { 
               "message" => "%{CLIENTIP:clientip}"
           }   
       }   
   }
   output {
       stdout {
           codec => rubydebug
       }   
   }
   

   调用结果

   {
         "sequence" => 0,
         "@version" => "1",
         "clientip" => "127.0.0.1",
          "message" => "127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] \"GET /favicon.ico HTTP/1.1\" 404 3650 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0\" \"-\"",
             "host" => "localhost.localdomain",
       "@timestamp" => 2020-11-16T10:18:12.671Z
   }
   

   2.2、直接在配置文件中调用时定义

   input {
       generator {
           count => 1
           message => '127.0.0.1 - - [04/Nov/2020:18:07:31 -0800] "GET /favicon.ico HTTP/1.1" 404 3650 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/    68.0" "-"'
       }   
   }
   filter {
       grok {
           match => { 
               "message" => "(?<client>\d{1,4}.\d{1,4}.\d{1,4}.\d{1,4})"
   						#定义格式为(?<name>pattern)
           }   
       }   
   }
   output {
       stdout {
           codec => rubydebug
       }   
   }
   

3、使用grok debug测试

grok debug使用方法