logtash 解析slow.log

LOST_9

已于 2022-08-22 15:42:24 修改

阅读量136

点赞数

分类专栏： logtash 文章标签： elk

于 2022-08-22 15:41:41 首次发布

本文链接：https://blog.youkuaiyun.com/u013091109/article/details/126467273

版权

logtash 专栏收录该内容

1 篇文章

订阅专栏

该配置文件描述了一个日志处理流程，从MySQL慢查询日志开始，通过Logstash进行过滤和解析，丢弃特定的SELECT SLEEP事件，然后将处理后的日志数据发送到Kafka主题`mysql-slowlog`。这个过程涉及到了日志分析、数据过滤、时间戳处理和日志输出到Kafka的步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

input {
  #stdin {
  file {
    type => "mysql-slow"
    path => "/data/mysqldata/log/slow.log"
    start_position => "end"
    codec => multiline {
      pattern => "# Time:"
      negate => true
      what => previous
          auto_flush_interval => 5
    }
  }
}

filter {
  # drop sleep events
  grok {
    match => { "message" => "SELECT SLEEP" }
    add_tag => [ "sleep_drop" ]
    tag_on_failure => [] # prevent default _grokparsefailure tag on real records
  }
  if "sleep_drop" in [tags] {
    drop {}
  }
  grok {
match => { "message" => "(?m)^#\s+Time:\s+%{GREEDYDATA:Time}#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?<clienthost>\S*) )?\[(?:%{IPV4:clientip})?\]\s+Id:\s+%{NUMBER:row_id:int}.*#\s+Query_time:\s+%{NUMBER:query_time:float}\s+Lock_time:\s+%{NUMBER:lock_time:float}\s+Rows_sent:\s+%{NUMBER:rows_sent:int}\s+Rows_examined:\s+%{NUMBER:rows_examined:int}.*\s*(?:use %{DATA:database};\s*)?SET\s+timestamp=%{NUMBER:timestamp};\s*(?<sql>(?<action>\w+)\b.*;)\s*(?:\n#\s+Time)?.*$" }

match => {"user" => "\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+"}

}
  date {
    match => [ "timestamp", "UNIX" ]
    remove_field => [ "timestamp" ]
  }


}

output {
  kafka {
    bootstrap_servers => "ip：端口,ip：端口,ip：端口"
    codec => json
    topic_id => "mysql-slowlog"
    }

  stdout { codec => rubydebug }
}