filebeat直连elasticsearch利用pipeline提取message中的字段

最新推荐文章于 2024-10-30 13:00:12 发布

转载最新推荐文章于 2024-10-30 13:00:12 发布 · 906 阅读

1 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/remainsu/p/filebeat-zhi-lianelasticsearch-li-yongp

文章标签：

#大数据

博客介绍了使用filebeat直连elasticsearch完成数据传输的方式。虽无logstash强大过滤功能，但因业务需求，仍需从原始数据中提取某些字段。具体步骤包括在/path/目录下建pipeline.json文件、上传规则到elasticsearch等，还给出了参考链接。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

这里使用filebeat直连elasticsearch的形式完成数据传输，由于没有logstash，所有对于原始数据的过滤略显尴尬（logstash的filter非常强大）。但是由于业务需求，还是需要将message（原始数据）中的某些字段进行提取，具体方式如下：

1. /path/目录下建立pipeline.json文件

{
  "description" : "test-pipeline",
  "processors" : [
    {
      "grok" :{
        "field" : "message",
        "patterns" : ["%{DATA:puid}\\\t%{DATA:datatime}\\\t\\\t%{DATA:content}"]
      }
    }
  ]
}

2. 将规则上传至elasticsearch中

curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_ingest/pipeline/test-pipeline' -d@/path/pipeline.json

3. filebeat.yml中

filebeat.prospectors:
 ******
 ******
output.elasticsearch:
  hosts: ["localhost:9200"]
  # 加入如下行：
  pipeline: "test-pipeline"

4. 测试数据

f1b25095cc823e63389ff299622b7e85    2019/02/27 03:38:54     send packet! opcode:3 message is in lua8282
f1b25095cc823e63389ff299622b7e85    2019/02/27 03:38:54     PacketManager::_onReceivedPacket opcode:3 size:27,rec_len:278282

5. elasticsearch中数据结果

[
    {
        "_score":1,
        "_type":"doc",
        "_id":"zWmLj2kB7ah0Pw2MmQGw",
        "_source":{
            "datatime":"2019/02/27 03:38:54",
            "log":{
                "file":{
                    "path":"/path/test_1.log"
                }
            },
            "beat":{
                "hostname":":",
                "name":":",
                "version":"6.6.1"
            },
            "@timestamp":"2019-03-18T06:44:43.224Z",
            "host":{
                "name":":"
            },
            "content":"",
            "source":"/path/test_1.log",
            "puid":"f1b25095cc823e63389ff299622b7e85",
            "offset":0,
            "input":{
                "type":"log"
            },
            "message":"f1b25095cc823e63389ff299622b7e85 2019/02/27 03:38:54 send packet! opcode:3 message is in lua",
            "prospector":{
                "type":"log"
            }
        },
        "_index":"test"
    },
    {
        "_score":1,
        "_type":"doc",
        "_id":"0GmLj2kB7ah0Pw2MmQGw",
        "_source":{
            "datatime":"2019/02/27 03:38:54",
            "log":{
                "file":{
                    "path":"/path/test_1.log"
                }
            },
            "beat":{
                "hostname":":",
                "name":":",
                "version":"6.6.1"
            },
            "@timestamp":"2019-03-18T06:44:43.224Z",
            "host":{
                "name":":"
            },
            "content":"",
            "source":"/path/test_1.log",
            "puid":"f1b25095cc823e63389ff299622b7e85",
            "offset":318,
            "input":{
                "type":"log"
            },
            "message":"f1b25095cc823e63389ff299622b7e85 2019/02/27 03:38:54 PacketManager::_onReceivedPacket| ReceivedPacket size:27",
            "prospector":{
                "type":"log"
            }
        },
        "_index":"test"
    }
]