day03-LogStash

最新推荐文章于 2025-06-19 11:12:25 发布
原创 最新推荐文章于 2025-06-19 11:12:25 发布 · 687 阅读 · 10 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#ELFK架构

LogStash环境搭建

1.下载Logstash 
[10:17:18 root@elk2:/usr/local]#wget  http://192.168.13.253/Resources/ElasticStack/softwares/logstash-7.17.23-amd64.deb


2.安装Logstash 
[10:17:35 root@elk2:/usr/local]#dpkg -i logstash-7.17.23-amd64.deb 


3.创建软连接，可以基于PATH变量访问logstash程序
[10:19:11 root@elk2:~]#ln -svf /usr/share/logstash/bin/logstash /usr/local/bin


4.查看帮助信息
#LogStash会启动一个jvm虚拟机所以会启动非常慢。
[10:20:39 root@elk2:~]#logstash -h
Using JAVA_HOME defined java: /usr/share/elasticsearch/jdk
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Usage:
    bin/logstash [OPTIONS]

Options:
    -n, --node.name NAME          Specify the name of this logstash instance, if no value is given
                                  it will default to the current hostname.
                                   (default: "elk2")
    --enable-local-plugin-development Allow Gemfile to be manipulated directly
                                  to facilitate simpler local plugin
                                  development.
                                  This is an advanced setting, intended
                                  only for use by Logstash developers,
                                  and should not be used in production.
                                   (default: false)
    -f, --path.config CONFIG_PATH Load the logstash config from a specific file
                                  or directory.  If a directory is given, all
                                  files in that directory will be concatenated
                                  in lexicographical order and then parsed as a
                                  single config file. You can also specify
                                  wildcards (globs) and any matched files will
                                  be loaded in the order described above.
    -e, --config.string CONFIG_STRING Use the given string as the configuration
                                  data. Same syntax as the config file. If no
                                  input is specified, then the following is
                                  used as the default input:
                                  "input { stdin { type => stdin } }"
                                  and if no output is specified, then the
                                  following is used as the default output:
                                  "output { stdout { codec => rubydebug } }"
                                  If you wish to use both defaults, please use
                                  the empty string for the '-e' flag.
                                   (default: nil)
    --field-reference-parser MODE (DEPRECATED) This option is no longer
                                  configurable.
                                  
                                  Use the given MODE when parsing field
                                  references.
                                  
                                  The field reference parser is used to expand
                                  field references in your pipeline configs,
                                  and has become more strict to better handle
                                  ambiguous- and illegal-syntax inputs.
                                  
                                  The only available MODE is:
                                   - `STRICT`: parse in a strict manner; when
                                     given ambiguous- or illegal-syntax input,
                                     raises a runtime exception that should
                                     be handled by the calling plugin.
                                  
                                   (default: "STRICT")
    --modules MODULES             Load Logstash modules.
                                  Modules can be defined using multiple instances
                                  '--modules module1 --modules module2',
                                     or comma-separated syntax
                                  '--modules=module1,module2'
                                  Cannot be used in conjunction with '-e' or '-f'
                                  Use of '--modules' will override modules declared
                                  in the 'logstash.yml' file.
    -M, --modules.variable MODULES_VARIABLE Load variables for module template.
                                  Multiple instances of '-M' or
                                  '--modules.variable' are supported.
                                  Ignored if '--modules' flag is not used.
                                  Should be in the format of
                                  '-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"'
                                  as in
                                  '-M "example.var.filter.mutate.fieldname=fieldvalue"'
    --setup                       Load index template into Elasticsearch, and saved searches, 
                                  index-pattern, visualizations, and dashboards into Kibana when
                                  running modules.
                                   (default: false)
    --cloud.id CLOUD_ID           Sets the elasticsearch and kibana host settings for
                                  module connections in Elastic Cloud.
                                  Your Elastic Cloud User interface or the Cloud support
                                  team should provide this.
                                  Add an optional label prefix '<label>:' to help you
                                  identify multiple cloud.ids.
                                  e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy'
    --cloud.auth CLOUD_AUTH       Sets the elasticsearch and kibana username and password
                                  for module connections in Elastic Cloud
                                  e.g. 'username:<password>'
    --pipeline.id ID              Sets the ID of the pipeline.
                                   (default: "main")
    -w, --pipeline.workers COUNT  Sets the number of pipeline workers to run.
                                   (default: 2)
    --pipeline.ordered ORDERED    Preserve events order. Possible values are `auto` (default), `true` and `false`.
                                  This setting
                                  will only work when also using a single worker for the pipeline.
                                  Note that when enabled, it may impact the performance of the filters
                                  and ouput processing.
                                  The `auto` option will automatically enable ordering if the
                                  `pipeline.workers` setting is set to `1`.
                                  Use `true` to enable ordering on the pipeline and prevent logstash
                                  from starting if there are multiple workers.
                                  Use `false` to disable any extra processing necessary for preserving
                                  ordering.
                                   (default: "auto")
    --java-execution              Use Java execution engine.
                                   (default: true)
    --plugin-classloaders         (Beta) Load Java plugins in independent classloaders to isolate their dependencies.
                                   (default: false)
    -b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in.
                                   (default: 125)
    -u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling
                                  for the next event.
                                   (default: 50)
    --pipeline.unsafe_shutdown    Force logstash to exit during shutdown even
                                  if there are still inflight events in memory.
                                  By default, logstash will refuse to quit until all
                                  received events have been pushed to the outputs.
                                   (default: false)
    --pipeline.ecs_compatibility STRING Sets the pipeline's default value for `ecs_compatibility`,
                                  a setting that is available to plugins that implement
                                  an ECS Compatibility mode for use with the Elastic Common
                                  Schema.
                                  Possible values are:
                                   - disabled (default)
                                   - v1
                                   - v2
                                  This option allows the early opt-in (or preemptive opt-out)
                                  of ECS Compatibility modes in plugins, which is scheduled to
                                  be on-by-default in a future major release of Logstash.
                                  
                                  Values other than `disabled` are currently considered BETA,
                                  and may produce unintended consequences when upgrading Logstash.
                                   (default: "disabled")
    --path.data PATH              This should point to a writable directory. Logstash
                                  will use this directory whenever it needs to store
                                  data. Plugins will also have access to this path.
                                   (default: "/usr/share/logstash/data")
    -p, --path.plugins PATH       A path of where to find plugins. This flag
                                  can be given multiple times to include
                                  multiple paths. Plugins are expected to be
                                  in a specific directory hierarchy:
                                  'PATH/logstash/TYPE/NAME.rb' where TYPE is
                                  'inputs' 'filters', 'outputs' or 'codecs'
                                  and NAME is the name of the plugin.
                                   (default: [])
    -l, --path.logs PATH          Write logstash internal logs to the given
                                  file. Without this flag, logstash will emit
                                  logs to standard output.
                                   (default: "/usr/share/logstash/logs")
    --log.level LEVEL             Set the log level for logstash. Possible values are:
                                    - fatal
                                    - error
                                    - warn
                                    - info
                                    - debug
                                    - trace
                                   (default: "info")
    --config.debug                Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled).
                                  WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result
                                  in plaintext passwords appearing in your logs!
                                   (default: false)
    -i, --interactive SHELL       Drop to shell instead of running as normal.
                                  Valid shells are "irb" and "pry"
    -V, --version                 Emit the version of logstash and its friends,
                                  then exit.
    -t, --config.test_and_exit    Check configuration for valid syntax and then exit.
                                   (default: false)
    -r, --config.reload.automatic Monitor configuration changes and reload
                                  whenever it is changed.
                                  NOTE: use SIGHUP to manually reload the config
                                   (default: false)
    --config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location
                                  for changes, in seconds.
                                   (default: #<Java::OrgLogstashUtil::TimeValue:0x291a8ecd>)
    --api.enabled ENABLED         Can be used to disable the Web API, which is
                                  enabled by default.
                                   (default: true)
    --api.http.host HTTP_HOST     Web API binding host (default: "127.0.0.1")
    --api.http.port HTTP_PORT     Web API http port (default: 9600..9700)
    --log.format FORMAT           Specify if Logstash should write its own logs in JSON form (one
                                  event per line) or in plain text (using Ruby's Object#inspect)
                                   (default: "plain")
    --path.settings SETTINGS_DIR  Directory containing logstash.yml file. This can also be
                                  set through the LS_SETTINGS_DIR environment variable.
                                   (default: "/usr/share/logstash/config")
    --verbose                     Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    --debug                       Set the log level to debug.
                                  DEPRECATED: use --log.level=debug instead.
    --quiet                       Set the log level to info.
                                  DEPRECATED: use --log.level=info instead.
    --http.enabled                Can be used to disable the Web API, which is
                                  enabled by default.
                                  DEPRECATED: use `--api.enabled=false`
    --http.host HTTP_HOST         Web API binding host
                                  DEPRECATED: use `--api.http.host=IP`
    --http.port HTTP_PORT         Web API http port
                                  DEPRECATED: use `--api.http.port=PORT`
    -h, --help                    print help

LogStash的架构

#logstash服务器:
指的是部署了Logstash环境的服务器。
	

#Logstash实例:
指的是运行了对应Logstash的进程。


#pipeline:
同一个Logstash实例可以有多个pipeline，如果未定义，则默认只有一个main pipeline。
	
同一个pipeline中有多个插件:
	- input:
		必须有，数据从哪来。
		
	- filter:
		该插件可选操作，数据做哪些处理，处理后再流向output插件。
		
	- output:
		必须有，数据到哪去。

LogStash的两种启动方式

`---------------------#基于命令行启动---------------------`
[10:21:39 root@elk2:~]#logstash -e input "input { stdin { type => stdin } }  output { stdout { codec => rubydebug } }"


#观察
[INFO ] 2024-10-29 10:34:13.751 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}


#测试。输入1111111111111111111
{
          "host" => "elk2",
          "type" => "stdin",
    "@timestamp" => 2024-10-29T02:36:28.784Z,
       "message" => "1111111111111111111",
      "@version" => "1"
}


#停止服务
ctrl+c终止

`---------------------#基于配置文件方式启动---------------------`
#参考方式一：
#当我们修改完配置后服务会热加载文件
[10:39:24 root@elk2:/etc/logstash/conf.d]#vim 01-stdin-to-stdout.conf
input {
  stdin { 
     type => stdin 
  }
}

output {
  stdout {
    codec => rubydebug
  }
}



[10:44:11 root@elk2:/etc/logstash/conf.d]#logstash -f 01-stdin-to-stdout.conf

#观察信息
[INFO ] 2024-10-29 10:44:36.810 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}

#输入信息测试：222222222222222
{
      "@version" => "1",
          "host" => "elk2",
    "@timestamp" => 2024-10-29T02:45:06.936Z,
          "type" => "stdin",
       "message" => "222222222222222"
}


#参考方式二：
[10:45:29 root@elk2:/etc/logstash/conf.d]#vim 01-stdin-to-stdout.conf
input {
  stdin { 
     type => stdin 
  }
}

output {
  stdout {
    #codec => rubydebug
     codec => json
  }
}


#观察信息
[INFO ] 2024-10-29 10:46:56.507 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}



#输入信息测试：333333333333333333333
{"host":"elk2","@version":"1","@timestamp":"2024-10-29T02:49:54.283Z","type":"stdin","message":"333333333333333333333"}

python脚本产生日志并使用Logstash采集

[11:10:43 root@elk2:~]#vim generate_log.py
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# @author : Jason Yin

import datetime
import random
import logging
import time
import sys

LOG_FORMAT = "%(levelname)s %(asctime)s [com.edu.%(module)s] - %(message)s "
DATE_FORMAT = "%Y-%m-%d %H:%M:%S"

# 配置root的logging.Logger实例的基本配置
logging.basicConfig(level=logging.INFO, format=LOG_FORMAT, datefmt=DATE_FORMAT, filename=sys.argv[1]
, filemode='a',)
actions = ["浏览页面", "评论商品", "加入收藏", "加入购物车", "提交订单", "使用优惠券", "领取优惠券",
 "搜索", "查看订单", "付款", "清空购物车"]

while True:
    time.sleep(random.randint(1, 5))
    user_id = random.randint(1, 10000)
    # 对生成的浮点数保留2位有效数字.
    price = round(random.uniform(15000, 30000),2)
    action = random.choice(actions)
    svip = random.choice([0,1,2])
    logging.info("DAU|{0}|{1}|{2}|{3}".format(user_id, action,svip,price))


    

#配置完后启动，就不要再动了，再开一个窗口去执行其他命令
[11:14:39 root@elk2:~]#python3 generate_log.py /tmp/apps.log   



#第二个窗口
[11:34:46 root@elk2:~]#cat /etc/logstash/conf.d/02-file-to-stdout.conf 
input { 
  file { 
     # 指定采集文件的路径
     path => "/tmp/apps.log"
     
 	 # 指定"首次"采集文件时从哪个位置读取: beginning, end(默认值)。
     start_position => "beginning"
  
  } 
}  

output { 
  stdout { 
    codec => rubydebug 
  } 
}


#第二个窗口启动
[11:37:04 root@elk2:~]#logstash -rf /etc/logstash/conf.d/02-file-to-stdout.conf



#第三个窗口查看我们的/tmp/apps.log
[11:38:32 root@elk2:~]#tail -f /tmp/apps.log 
INFO 2024-10-29 11:38:12 [com.generate_log] - DAU|843|加入购物车|2|28042.12 
INFO 2024-10-29 11:38:17 [com.generate_log] - DAU|6363|领取优惠券|0|20939.4 
INFO 2024-10-29 11:38:19 [com.generate_log] - DAU|6314|清空购物车|2|18861.92 
.........................

LogStash的filter插件基本使用

#插件汇总
mutate
date
grok
useragent
geoip

mutate插件分析自定义app日志案例

#注意：第一个窗口的脚本别关
[12:00:23 root@elk2:~]#python3 generate_log.py /tmp/apps.log


1.编写配置文件
[12:01:28 root@elk2:~]#vim /etc/logstash/conf.d/03-file-mutate-stdout.conf 
 
input { 
  file { 
     path => "/tmp/apps.log"

     start_position => "beginning"
  } 
}  


filter {
  mutate {
    # 将message字段按照"|"进行切割
    split => { "message" => "|" }

    # 添加字段
    add_field => { 
       "userId" => "%{[message][1]}"
       "action" => "%{[message][2]}"
       "svip" => "%{[message][3]}"
       "price" => "%{[message][4]}"
       "other" => "%{[message][0]}"
    }

    # 移除字段
    remove_field => [ "message","@version","host" ]

  }

  mutate {
    # 转换字段的数据类型
    convert => {
      "userId" => "integer"
      "price" => "float"
    }
  }

}


output { 
  stdout { 
    codec => rubydebug 
  } 
}


2.启动Logstash实例
[12:04:31 root@elk2:/etc/logstash/conf.d]#logstash -rf 03-file-mutate-stdout.conf 


3.第二次窗口，验证测试
查看2启动后出现的信息

date插件准备处理日期格式案例

#注意：第一个窗口的脚本别关
[12:00:23 root@elk2:~]#python3 generate_log.py /tmp/apps.log

1.编写配置文件
[15:06:53 root@elk2:/etc/logstash/conf.d]#vim 03-file-mutate-stdout.conf 
input { 
  file { 
     path => "/tmp/apps.log"

     start_position => "beginning"
  } 
}  


filter {
  mutate {
    # 将message字段按照"|"进行切割
    split => { "message" => "|" }

    # 添加字段
    add_field => { 
       "userId" => "%{[message][1]}"
       "action" => "%{[message][2]}"
       "svip" => "%{[message][3]}"
       "price" => "%{[message][4]}"
       "other" => "%{[message][0]}"
    }

  }

  mutate {
    # 转换字段的数据类型
    convert => {
      "userId" => "integer"
      "price" => "float"
    }
  }


  mutate {
    split => { "other" => " " }

    add_field => { 
       "dt" => "%{[other][1]} %{[other][2]}"
    }

    # 移除字段
    remove_field => [ "message","@version","host","other" ]
  }


  date {
    # 2024-10-29 14:43:21
    # 语法参考:
    #   https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match
    match => [ "dt", "yyyy-MM-dd HH:mm:ss" ]

    # 将转换的数据存储在指定的字段中，若不指定，则默认会覆盖"@timestamp"字段
    #target => "datetime-li"

    # 定义时区的配置,参考地址:
    #  https://joda-time.sourceforge.net/timezones.html
    timezone => "Asia/Shanghai"
  }

}


output { 
  stdout { 
    codec => rubydebug 
  } 

  elasticsearch {
      # 指定ES集群的地址
      hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]
      # 指定ES集群的索引名称
      index => "linux-logstash-%{+yyyy.MM.dd}"
  }

}



2.启动Logstash实例 
[15:13:59 root@elk2:/etc/logstash/conf.d]#sed  -i 's#2024#2023#g' /tmp/apps.log 
[root@elk92 conf.d]# rm -rf /usr/share/logstash/data/
[15:14:02 root@elk2:/etc/logstash/conf.d]#rm -rf /usr/share/logstash/data/
[15:14:14 root@elk2:/etc/logstash/conf.d]#
 logstash -rf 03-file-mutate-stdout.conf 


#这个脚本是我又执行了一遍，为了生成新数据
[15:17:05 root@elk2:~]#python3 generate_log.py /tmp/apps.log
	
	
3.kibana查询数据
先看索引管理-->创建索引-->去Discover中查看-->点击创建的索引

在这里插入图片描述

在这里插入图片描述

Kibana分析数据（画图）

创建聚合索引

在这里插入图片描述

视图库创建UV

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

ELFK-用户行为统计

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

ELFK-平台交易额

在这里插入图片描述

在这里插入图片描述

创建仪表盘（Dashboard）

在这里插入图片描述

在这里插入图片描述

Logstash对接Filebeat

1.编写Logstash配置文件 
[16:17:14 root@elk2:/etc/logstash/conf.d]#vim 04-beats-grok-stdout.conf
input { 
  beats {
    # 监听的端口号
    port => 8888
  }

}  

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

}

output { 
  stdout { 
    codec => rubydebug 
  } 
}

	
	

2.启动Logstash 
[16:28:59 root@elk2:/etc/logstash/conf.d]# logstash -rf 04-beats-grok-stdout.conf
...	
[INFO ] 2024-10-29 16:29:45.880 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:8888"}
...



3.编写filebeat配置文件 
[16:29:59 root@elk3:/etc/filebeat]#vim 09-nginx-to-logstash.yam
filebeat.inputs:
- type: log
  paths:
  - /var/log/nginx/access.log*

# 输出地址为Logstash的8888端口
output.logstash:
  hosts: ["10.0.0.92:8888"]




4.启动filebeat采集数据到Logstash节点
[16:30:23 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[16:30:45 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml

基于grok分析nginx的访问日志

#第一个窗口
[20:24:48 root@elk2:/etc/logstash/conf.d]#logstash -rf 04-beats-grok-stdout.conf

1.编写Logstash配置文件（第二个窗口）
[20:06:53 root@elk2:/etc/logstash/conf.d]#vim 04-beats-grok-stdout.conf 
input { 
  beats {
    # 监听的端口号
    port => 8888
  }

}  

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

  # 基于正则提取任意文本
  grok {
     # 分析message字段的访问日志，基于内置的变量: HTTPD_COMMONLOG
     match => { "message" => "%{HTTPD_COMMONLOG}"}
   }


}


output { 
  stdout { 
    codec => rubydebug 
  } 
}



2.启动filebeat 
[19:06:16 root@elk3:~]#rm -rf /var/lib/filebeat/
[20:11:03 root@elk3:/etc/filebeat/conf.d]#filebeat -e -c 09-nginx-to-logstash.yaml 



3.测试数据效果：
{
           "verb" => "GET",
          "ident" => "-",
        "request" => "/",
          "bytes" => "396",
     "@timestamp" => 2024-10-29T12:52:45.683Z,
       "clientip" => "221.218.209.11",
      "timestamp" => "28/Oct/2024:17:11:50 +0800",
           "host" => {
        "name" => "elk3"
    },
           "auth" => "-",
    "httpversion" => "1.1",
       "response" => "200",
        "message" => "221.218.209.11 - - [28/Oct/2024:17:11:50 +0800] \"GET / HTTP/1.1\" 200 396 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15\""
}

基于geoip分析nginx的IP地址经纬度

#第一个窗口
[20:36:57 root@elk2:/etc/logstash/conf.d]#logstash -rf 05-beats-grok-stdout.conf
......
[INFO ] 2024-10-29 20:37:38.375 [[main]<beats] Server - Starting server on port: 8888
.....




1.编写Logstash配置文件
[20:26:50 root@elk2:/etc/logstash/conf.d]#vim 05-beats-geoip-stdout.conf 

input { 
  beats {
    port => 8888
  }

}  

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

  grok {
     match => { "message" => "%{HTTPD_COMMONLOG}"}
   }

  # 基于指定公网IP地址字段分析用户的经纬度，城市，国家等相关信息。
  geoip {
    source => "clientip"
  }

}


output { 
  stdout { 
    codec => rubydebug 
  } 
}


2.filebeat启动实例 
[20:27:37 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[20:27:40 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml 



3.去第一个窗口查看效果
{
     "@timestamp" => 2024-10-29T12:59:48.839Z,
       "clientip" => "150.88.66.22",
       "response" => "200",
          "bytes" => "396",
           "auth" => "-",
           "verb" => "GET",
      "timestamp" => "28/Oct/2024:17:12:27 +0800",
        "request" => "/iphone",
           "host" => {
        "name" => "elk3"
    },
          "geoip" => {
             "longitude" => 139.6895,
         "country_code2" => "JP",
        "continent_code" => "AS",
              "location" => {
            "lon" => 139.6895,
            "lat" => 35.6897
        },
                    "ip" => "150.88.66.22",
         "country_code3" => "JP",
          "country_name" => "Japan",
              "latitude" => 35.6897,
              "timezone" => "Asia/Tokyo"
    },
          "ident" => "-",
        "message" => "150.88.66.22 - - [28/Oct/2024:17:12:27 +0800] \"GET /iphone HTTP/1.1\" 200 396 \"-\" \"Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36\"",
    "httpversion" => "1.1"
}

基于useragent分析用户的设备类型

1.编写Logstash配置文件
[20:59:29 root@elk2:/etc/logstash/conf.d]#vim 06-beats-useragent-stdout.conf

input { 
  beats {
    port => 8888
  }

}  

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

  grok {
     match => { "message" => "%{HTTPD_COMMONLOG}"}
   }

  geoip {
    source => "clientip"
  }

  useragent {
    # 基于哪个字段分析用户的设备类型
    source => "message"
   
    # 将分析的结果存储在哪个字段中
    target => "linux-useragent"
  }

}


output { 
  stdout { 
    codec => rubydebug 
  } 
}


[21:03:34 root@elk2:/etc/logstash/conf.d]#logstash -rf 06-beats-useragent-stdout.conf



2.filebeat启动实例
[21:04:12 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[21:04:14 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml 



3.测试结果
{
    "linux-useragent" => {
        "os_version" => "8.0.0",
             "major" => "116",
          "os_major" => "8",
             "patch" => "0",
                "os" => "Android",
            "device" => "Samsung SM-G955U",
           "os_name" => "Android",
           "os_full" => "Android 8.0.0",
          "os_patch" => "0",
             "minor" => "0",
           "version" => "116.0.0.0",
          "os_minor" => "0",
              "name" => "Chrome Mobile"
    },
                     "@timestamp" => 2024-10-29T13:04:43.443Z,
                       "response" => "200",
                        "request" => "/iphone",
                           "auth" => "-",
                          "bytes" => "396",
                       "clientip" => "150.88.66.22",
                          "ident" => "-",
                           "host" => {
        "name" => "elk3"
    },
                           "verb" => "GET",
                    "httpversion" => "1.1",
                          "geoip" => {
             "longitude" => 139.6895,
         "country_code3" => "JP",
              "timezone" => "Asia/Tokyo",
              "latitude" => 35.6897,
        "continent_code" => "AS",
         "country_code2" => "JP",
          "country_name" => "Japan",
                    "ip" => "150.88.66.22",
              "location" => {
            "lat" => 35.6897,
            "lon" => 139.6895
        }
    },
                      "timestamp" => "28/Oct/2024:17:12:27 +0800",
                        "message" => "150.88.66.22 - - [28/Oct/2024:17:12:27 +0800] \"GET /iphone HTTP/1.1\" 200 396 \"-\" \"Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36\""
}

基于date分析nginx的访问时间

1.编写Logstash配置文件
[21:04:41 root@elk2:/etc/logstash/conf.d]#vim 07-beats-date-stdout.conf

input { 
  beats {
    port => 8888
  }

}

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

  grok {
     match => { "message" => "%{HTTPD_COMMONLOG}"}
   }

  geoip {
    source => "clientip"
  }

  useragent {
    source => "message"
   
    target => "linux-useragent"
  }

  date {
    # 28/Oct/2024:17:12:27 +0800
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }

}

output { 
  stdout { 
    codec => rubydebug 
  } 
}


[21:06:25 root@elk2:/etc/logstash/conf.d]#logstash -rf 07-beats-useragent-stdout.conf 



2.filebeat启动实例
[21:04:12 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[21:04:14 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml 




3.测试数据效果：
{
                       "response" => "200",
                      "timestamp" => "28/Oct/2024:17:12:27 +0800",
                    "httpversion" => "1.1",
                           "auth" => "-",
                          "geoip" => {
        "continent_code" => "AS",
             "longitude" => 139.6895,
         "country_code3" => "JP",
         "country_code2" => "JP",
              "location" => {
            "lon" => 139.6895,
            "lat" => 35.6897
        },
              "timezone" => "Asia/Tokyo",
                    "ip" => "150.88.66.22",
              "latitude" => 35.6897,
          "country_name" => "Japan"
    },
                           "verb" => "GET",
                          "bytes" => "396",
    "linux-useragent" => {
           "os_name" => "Android",
             "minor" => "0",
          "os_patch" => "0",
             "major" => "116",
           "os_full" => "Android 8.0.0",
          "os_minor" => "0",
          "os_major" => "8",
             "patch" => "0",
                "os" => "Android",
           "version" => "116.0.0.0",
        "os_version" => "8.0.0",
            "device" => "Samsung SM-G955U",
              "name" => "Chrome Mobile"
    },
                       "clientip" => "150.88.66.22",
                     "@timestamp" => 2024-10-28T09:12:27.000Z,
                          "ident" => "-",
                           "host" => {
        "name" => "elk3"
    },
                        "request" => "/iphone",
                        "message" => "150.88.66.22 - - [28/Oct/2024:17:12:27 +0800] \"GET /iphone HTTP/1.1\" 200 396 \"-\" \"Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Mobile Safari/537.36\""
}

将数据写入到ES集群

1.编写Logstash配置文件
[21:07:37 root@elk2:/etc/logstash/conf.d]#vim 08-beats-filter-es.conf

input { 
  beats {
    port => 8888
  }

}  

filter {
  mutate {
    remove_field => [ "@version","log","tags","agent","ecs","input" ]
  }

  grok {
     match => { "message" => "%{HTTPD_COMMONLOG}"}
   }

  geoip {
    source => "clientip"
  }

  useragent {
    source => "message"
   
    target => "linux-useragent"
  }

  date {
    # 28/Oct/2024:17:12:27 +0800
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }

}


output { 
  stdout { 
    codec => rubydebug 
  } 

  elasticsearch {
      hosts => ["10.0.0.91:9200","10.0.0.92:9200","10.0.0.93:9200"]
      index => "linux-elfk-nginx-%{+yyyy.MM.dd}"
  }

}



[21:10:04 root@elk2:/etc/logstash/conf.d]#logstash -rf 08-beats-filter-es.conf



2.filebeat启动实例
[21:10:21 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[21:10:22 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml 



3.kibana查询数据并验证
#重新采集一下
[21:10:21 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[21:10:22 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml

在这里插入图片描述

ELFK架构数据映射之地图故障案例讲解

在这里插入图片描述

在这里插入图片描述

解决方法：

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

{
  "number_of_replicas": 0,
  "number_of_shards": 6
}

在这里插入图片描述

在这里插入图片描述

在这里插入图片描述

[21:40:37 root@elk3:/etc/filebeat]#rm -rf /var/lib/filebeat/
[21:40:39 root@elk3:/etc/filebeat]#filebeat -e -c 09-nginx-to-logstash.yaml

在这里插入图片描述

删除索引

在这里插入图片描述

在这里插入图片描述

重新创建索引

在这里插入图片描述

故障修复

在这里插入图片描述

在这里插入图片描述

统计IP

统计UV

地理位置

设备类型

在这里插入图片描述

今日总结

- Logstash的环境搭建
- logstash的架构
	- logstash实例
	- logstash的pipeline
		- input 
		- filter 
		- output
		
- Logstash常用的插件
	- input :
		- stdin 
		- beats 
		- file 
		
	- filter:
		- mutate
		- date 
		- grok
		- geoip
		- useragent
		
	- output
		- stdout 
		- elasticsearch
		
- ELK架构 

- ELFK架构

- 故障案例分享之数据映射
LogStash下载安装（Windows）
mynameisjinxiaokai的博客
09-07 3154
https://elasticsearch.cn/download/#seg-3
Day06-filebeat，logstash多实例，pipline，ElasticStack项目架构梳理及实战案例
苦难带不走梦想
07-16 722
02-写入地理位置-lat代表纬度,lon代表经度。filebeat的modules实战案例-了解。01-修改nginx的索引的地理位置映射。OLDBOYedu2023 教室07。完成的课堂的所有练习并整理思维导图.(2)修改pipline的配置文件。(3)启动logstash实例。(1)filebeat采集日志。(2)logstash分析数据。02-批量创建测试地理位置数据。(3)kibana出图展示。logstash配置文件。将数据发送到目的端。
Logstash下载与安装
MissOfSpring的博客
07-01 768
Logstash下载与安装 1.下载地址 下载地址 2.解压 解压至任何自己喜欢的目录。 需要注意的是logstash需要和Elasticsearch安装版本号一致。避免不必要的bug。 3. 运行 运行Logstash，可以参照官方的步骤 操作如下： C:\Windows\system32> cd C:\logstash-7.8.0 C:\logstash-7.8.0> .\bin\logstash.bat -f .\config\syslog.conf 我的实际上是在bin下创建了mys
logstash 配置
最新发布
技术博客共享
06-19 240
本文提供了两个Logstash配置示例。第一个配置实现TCP服务器功能，监听4567端口接收JSON格式日志数据，并输出到Elasticsearch的"sys_log"索引和标准输出。第二个配置实现MySQL数据库同步功能，每分钟从mv_system数据库的sys_log_operation表同步数据到Elasticsearch的"operate_info"索引，使用ID作为文档标识，同时支持分页查询和数据增量同步。两个配置都包含UTF-8编码处理和输出调试功能。
logstash-6.2.4.tar.gz
05-15
logstash-6.2.4.tar.gz elk 组件之一开源实时日志分析ELK平台能够完美的解决我们上述的问题，ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成
logstash-5.6.3.tar.gz
10-26
logstash-5.6.3.tar.gz 重量级日志收集、过滤、筛选的工具
logstash-7.1.1.zip
09-06
Logstash是一个开源的日志收集管理工具。... logstash作为一个数据管道中间件,支持对各种类型数据的采集与转换,并将数据发送到各种类型的存储库
Linux运维-04-日志分析-日志监控ELK-day03-生产案例及G
11-02
"Linux运维-04-日志分析-日志监控ELK-day03-生产案例及Git版本控制"这个主题深入探讨了如何使用ELK（Elasticsearch、Logstash、Kibana）堆栈进行日志管理和分析，并结合Git进行版本控制，以提升工作效率。...
day07-Elasticsearch03
12-13
在本课程"day07-Elasticsearch03"中，我们将深入探讨Elasticsearch这一强大的分布式搜索引擎和数据分析工具。Elasticsearch（ES）是基于Lucene构建的，它以其高效、可扩展性和实时分析能力而备受青睐。在这个阶段的...
云端技术驾驭DAY04——Logstash安装部署及插件模块
川川野马的博客
02-17 1336
什么是Logstash– 是一个数据采集、加工处理以及传输的工具Logstash特点– 所有类型的数据集中处理– 不同模式和格式数据的正常化– 自定义日志格式的迅速扩展– 为自定义数据源轻松添加插件Logstash结构原理结构解释input负责收集数据filter负责处理数据output负责输出数据Logstash支持的数据类型– 布尔、字节、字符串、数值、数组、键值对Logstash支持的判断方法判断方法解释==等于!不等于大于小于。
day05-Elasticsearch01day05-Elasticsearch01
10-25
在“day05-Elasticsearch01day05-Elasticsearch01”这个主题中，我们可以深入探讨Elasticsearch的基础概念、核心特性以及常见应用场景。 一、Elasticsearch基础 1. 数据模型：Elasticsearch基于Lucene构建，使用倒...
logstash-6.6.0.tar.gz
06-10
logstash-6.6.0
logstash-6.2.4.zip
07-02
logstash-6.2.4.zip， logstash-6.2.4.zip，logstash-6.2.4.zip
logstash-6.5.0.tar.gz
11-21
logstash-6.5.0.tar.gz，linux系统，官网资源，真实有效，欢迎下载
logstash-5.3.1 下载
05-01
logstash-5.3.1 资源下载
logstash安装包
08-30
logstash安装包，最新版
logstash下载安装
我不是蒸鱼的博客
11-08 2562
1.下载logstash https://www.elastic.co/cn/downloads/past-releases#logstash 2.解压 tar -zxvf logstash-6.7.0.tar.gz 3.测试是否启动正常 bin/logstash -e 'input { stdin {} } output { stdout {} }' 启动后 输入字符 控...
logstash安装
DT-TK
02-20 235
Logstash介绍 Logstash是一个开源的服务器端数据处理管道，可以同时从多个数据源获取数据，并对其进行转换，然后将其发送到你最喜欢的“存储”。（当然，我们最喜欢的是Elasticsearch） 下载Logstash 我的是 7.3.0 https://artifacts.elastic.co/downloads/logstash/logstash-7.3.0.zip 解压后安装...
ELK-logstash安装下载
q294761325的博客
06-03 263
ELK-logstash 安装下载 logstash-6.7.0.tar https://www.newbe.pro/Mirrors/Mirrors-Logstash/ 链接：https://pan.baidu.com/s/1HrtwbdEsuOjoE1t7oTSRkw 提取码：lobc 2.解压 tar -zxvf logstash-6.7.0.tar.gz 3.修改配置 在config文件夹中创建logstash-log.conf配置文件 input { beats {
Logstash java
01-22
pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY}" # 假设日期作为新记录的开头 negate => true what => previous } } } ``` 上述配置中的`codec.multiline`部分定义了一个模式用于识别新的日志项，并指示如果某...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值