logstash 例子

最新推荐文章于 2024-06-24 20:09:29 发布

原创最新推荐文章于 2024-06-24 20:09:29 发布 · 968 阅读

0 ·

CC 4.0 BY-SA版权

【构建高可用架构】同时被 2 个专栏收录

347 篇文章

订阅专栏

ELK

4 篇文章

订阅专栏

本文详细介绍使用Logstash进行多种类型日志收集的实际操作案例，包括Tomcat、Nginx日志及Syslog等，并提供了配置示例。此外还介绍了如何通过正则表达式解析日志、使用Redis缓存日志等高级技巧。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

logstash 例子

概述

1、收集tomcat日志一

使用 [ 作为日志切割

 input {  

	file { 
		type => "es_error"
        path => "E:/install/tempTwo/*.log"
		start_position => beginning
		codec => multiline {
			pattern => "^\["
			negate => true
			what => "previous"
		}
    }  
	
    file { 
		type => "tomcat_error"
        path => "E:/install/temp/*.log"
		start_position => beginning
		codec => multiline {
			pattern => "^\["
			negate => true
			what => "previous"
		}
    }  
} 


output {   

	if [type]  == "es_error" {
	    elasticsearch { hosts => localhost   
                    index => "logstash-es-%{+YYYY.MM.dd}"}  
	}
	
	if [type]  == "tomcat_error" {
	    elasticsearch { hosts => localhost   
                    index => "logstash-tomcat-%{+YYYY.MM.dd}"}  
	}

      
    stdout { codec => rubydebug  }   
}

2、收集tomcat日志二

input {  
    file { 
		type => "tomcat_error"
        path => "E:/install/temp/*.log"
		start_position => beginning
    }  
} 

filter {
    if [type] == "tomcat_error" {
            multiline {
                      pattern => "^[^\[]"
                      what => "previous"
                  }
                mutate {
           split => ["message", "|"]
        }
        grok {
            match => { 
                           "message" => "(?m)%{TIMESTAMP_ISO8601:logtime}"
            }
        }
    }
}
 

output {    
    elasticsearch { hosts => localhost   
                    index => "logstash-cmis-%{+YYYY.MM.dd}"}  
      
    stdout { codec => rubydebug  }   
}

3、收集nginx日志

设置nginx 的日志格式为json

input {  

	file { 
		type => "access_log"
        path => "E:/install/tempTwo/*.log"
		codec => json
		start_position => beginning
    }  
	 
}

4、收集syslog日志

logstash 监听tcp udp的514 端口
vim /etc/rsyslog.conf 最后增加

*.* @@192.168.0.11:514

input {  
	syslog { 
		type => "sys_log"
        host => "192.168.0.1"
		port => "514"
    } 
}

4、收集tcp日志

input {  
	tcp { 
		type => "tcp_log"
        host => "192.168.0.11"
		port => "6666"
    } 
}

安装 nc

nc 192.168.56.11 6666 < /etc/resolv.conf

收集slowlog-grok

常用表达式

缺点占用cpu

logstash-patterns-core

https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns


https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns