Common Log Formats and How To Parse Them

最新推荐文章于 2024-03-10 00:11:09 发布

转载最新推荐文章于 2024-03-10 00:11:09 发布 · 297 阅读

elk 专栏收录该内容

23 篇文章

订阅专栏

本文介绍如何使用Fluentd的Tail插件解析常见的日志格式，包括Apache和Nginx的日志，通过正则表达式匹配关键字段，实现高效的数据收集。

This page is a glossary of common log formats that can be parsed with the Tail input plugin.

Apache Access Log

Use format apache2 as shown below:

  <source>
      @type tail
      format apache2
      tag apache.access
      path /var/log/apache2/access.log
  </source>

Apache Error Log
Use a regular expression. See the format field in the following sample configuration.
```
  <source>
      @type tail
      format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] \[pid (?<pid>[^\]]*)\] \[client (?<client>[^\]]*)\] (?<message>.*)$/
      tag apache.error
      path /var/log/apache2/error.log
  </source>
```
Depending on your particular error log format, you may need to adjust the regular expression above. You can test your format using fluentd-ui’s in_tail editor or Fluentular.

Maillog

Use a regular expression. See the format field in the following sample configuration.

  <source>
      @type tail
      format /^(?<time>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
      tag postfix.maillog
      path /var/log/maillog
  </source>

Nginx Access Log

Use format nginx as shown below:

  <source>
      @type tail
      format nginx
      tag nginx.access
      path /var/log/nginx/access.log
  </source>

Nginx Error Log

Use the format* and multiline_flush_interval fields in the following sample configuration. Applications running under Nginx can output multi-line errors including stack traces, so the multiline mode is a good fit.

  <source>
      @type tail
      tag nginx.error
      path /var/log/nginx/error.log

      format multiline
      format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /
      format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/
      multiline_flush_interval 3s
  </source>

If you know your error log will only contain single lines, you can use the below simpler configuration with just a format.

  <source>
      @type tail
      format /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)$/
      tag nginx.error
      path /var/log/nginx/error.log
  </source>