LogStash的CodeC插件实现多行合并

Multiline codec pluginedit

  • Plugin version: v3.0.9
  • Released on: 2018-01-17
  • Changelog

Getting Helpedit

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Descriptionedit

The multiline codec will collapse multiline messages and merge them into a single event.

Important

If you are using a Logstash input plugin that supports multiple hosts, such as the Beats input plugin input plugin, you should not use the multiline codec to handle multiline events. Doing so may result in the mixing of streams and corrupted event data. In this situation, you need to handle multiline events before sending the event data to Logstash.

The original goal of this codec was to allow joining of multiline messages from files into a single event. For example, joining Java exception and stacktrace messages into a single event.

The config looks like this:

input {
  stdin {
    codec => multiline {
      pattern => "pattern, a regexp"
      negate => "true" or "false"
      what => "previous" or "next"
    }
  }
}

The pattern should match what you believe to be an indicator that the field is part of a multi-line event.

The what must be previous or next and indicates the relation to the multi-line event.

The negate can be true or false (defaults to false). If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true)

For example, Java stack traces are multiline and usually have the message starting at the far-left, with each subsequent line indented. Do this:

input {
  stdin {
    codec => multiline {
      pattern => "^\s"
      what => "previous"
    }
  }
}

This says that any line starting with whitespace belongs to the previous line.

Another example is to merge lines not starting with a date up to the previous line..

input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      # Grok pattern names are valid! :)
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => "previous"
    }
  }
}

This says that any line not starting with a timestamp should be merged with the previous line.

One more common example is C line continuations (backslash). Here’s how to do that:

input {
  stdin {
    codec => multiline {
      pattern => "\\$"
      what => "next"
    }
  }
}

This says that any line ending with a backslash should be combined with the following line.

Logstash Aggregate 插件Logstash 中的一个功能强大的插件,用于聚合和处理事件流数据。它可以将多个事件聚合成一个事件,并且可以根据指定的条件对事件进行分组、计算和转换。 Aggregate 插件通过使用一个或多个字段的值来标识要聚合的事件,并在指定的时间窗口内将它们合并为一个单独的事件。这个插件非常有用,特别是在处理日志数据时,可以将多行日志合并成一个完整的事件。 使用 Aggregate 插件需要配置一些参数,例如: - `task_id`:用于唯一标识每个聚合任务的ID。 - `code`:用于定义聚合逻辑的 Ruby 代码块。 - `map_action`:用于指定如何处理每个匹配的事件。 - `end_of_task`:用于定义何时结束聚合任务。 以下是一个简单的示例配置,演示如何使用 Aggregate 插件来处理多行日志: ```ruby input { stdin {} } filter { aggregate { task_id => "%{host}" code => " map['message'] ||= [] map['message'] << event.get('message') event.cancel() " map_action => "create" end_of_task => true } } output { stdout { codec => rubydebug } } ``` 在这个示例中,我们从标准输入获取日志数据,然后使用 Aggregate 插件来聚合多行日志。通过指定 `task_id` 为 `%{host}`,我们可以根据日志中的主机名来标识聚合任务。在 `code` 中,我们将每个事件的 `message` 字段添加到一个数组中,并取消原始事件的处理。最后,通过设置 `map_action` 为 "create" 和 `end_of_task` 为 true,我们告诉插件在聚合完成后创建一个新的事件,并输出到标准输出。 这只是一个简单的示例,你可以根据实际需求和数据结构来配置更复杂的聚合逻辑。Logstash 的官方文档中有更详细的说明和示例,你可以参考官方文档来深入了解 Logstash Aggregate 插件的用法。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值