ELK集群搭建及logstash+kibana展示--logstash

最新推荐文章于 2025-05-30 15:59:44 发布

原创最新推荐文章于 2025-05-30 15:59:44 发布 · 929 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#ELK #Logstash #kibana #elasticsearch #ELK集群

系统同时被 2 个专栏收录

11 篇文章

订阅专栏

ELK集群

3 篇文章

订阅专栏

官网地址：https://www.elastic.co/cn/downloads

下载链接，最新版的6.5.1。

Elasticsearch Kibana Logstash 记得版本保持一致

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.1.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.1.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.1-linux-x86_64.tar.gz

机器list

ip	logstash	elasticsearch	Kibana
192.168.3.17	√	√	√
192.168.3.16	×	√	×
192.168.3.18	×	√	×

解压logstash

tar zxvf src/logstash-6.5.1.tar.gz
vim logstash-6.5.1/config/std_test.conf            //新建测试conf
input {
    stdin{
    }
} 

output {
    stdout{
    }
}

测试

./logstash-6.5.1/bin/logstash -f logstash-6.5.1/config/std_test.conf        //启动测试，如果出现下列字符就表示安装安装成功

Sending Logstash logs to /home/yx/ma/logstash-6.5.1/logs which is now configured via log4j2.properties
[2018-11-28T17:38:43,938][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/yx/ma/logstash-6.5.1/data/queue"}
[2018-11-28T17:38:43,957][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/yx/ma/logstash-6.5.1/data/dead_letter_queue"}
[2018-11-28T17:38:44,791][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-28T17:38:44,813][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.1"}
[2018-11-28T17:38:44,866][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"469786ff-14c5-4b35-acc9-087b5e2ee47f", :path=>"/home/yx/ma/logstash-6.5.1/data/uuid"}
[2018-11-28T17:38:51,366][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-11-28T17:38:51,561][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4af52480 run>"}
The stdin plugin is now waiting for input:
[2018-11-28T17:38:51,664][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-11-28T17:38:52,103][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

新建调用nginx日志的配置文件

input {
	file{
	path =>"/home/yx/server/nginx/logs/*.log"            #通配符*号
	start_position=>"beginning"
	}
}

filter{
	grok{
		patterns_dir => "*.log"                            #通配符*号
		match=>{"message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""                #具体各项配置详解可参考链接：http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html
		}
		remove_field=>"message"
	}
	date{
		match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
	}
}
output {
    stdout{
    	codec=>rubydebug
    }
}

新建调用PHP日志的配置文件，只有input的输入路径不同，其他都一样。

input {
	file{
	path =>"/home/yx/server/php56/log/php-error.log"        #单个日志例子
	start_position=>"beginning"
	}
}

filter{
	grok{
		patterns_dir => "*.log"
		match=>{"message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""
		}
		remove_field=>"message"
	}
	date{
		match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
	}
}
output {
    stdout{
    	codec=>rubydebug
    }
}

指定多配置文件启动，个人感觉也可以将所有需显示的日志软连接到一个单独目录里，然后配置文件只写一个就好。

./logstash-6.5.1/bin/logstash -f logstash-6.5.1/config/std_nginx.conf -f logstash-6.5.1/config/std_php.conf     //前台启动，可以实时查看日志输出。
nohup ./logstash-6.5.1/bin/logstash -f logstash-6.5.1/config/std_nginx.conf -f logstash-6.5.1/config/std_php.conf & //后台启动并且不会影响输出。

此处可以另开两个窗口，tail -f 查看nginx的日志和PHP的日志，从而和logstash输出的日志做对比。

如果没有意外，到此logstash已经安装成功。请看第二篇