ELK + Filebeat + Kafka 分布式日志管理平台搭建

paths:

• /var/logs/springboot/sparksys-oauth.log

fields:

log_source: oauth

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

Array of hosts to connect to.

hosts: [“192.168.3.3:9200”]

#----------------------------- Logstash output --------------------------------

#output.logstash:

The Logstash hosts

hosts: [“logstash:5044”]

#----------------------------- kafka output --------------------------------

output.kafka:

enabled: true

hosts: [“192.168.3.3:9092”]

topic: sparksys-log

添加kafka输出的配置，将logstash输出配置注释掉。hosts表示kafka的ip和端口号，topic表示filebeat将数据输出到topic为sparksys-log的主题下，此处也根据自己情况修改

2.1.2 Logstash变动

====================================================================================

logstash.conf配置input由原来的输入源beat改为kafka

input {

kafka {

codec => “json”

topics => [“sparksys-log”]

bootstrap_servers => “192.168.3.3:9092”

auto_offset_reset => “latest”

group_id => “logstash-g1”

}

}

output {

elasticsearch {

hosts => “es:9200”

index => “filebeat_%{[fields][log_source]}-%{+YYYY.MM.dd}”

}

}

上述配置说明如下:

• topics后面的sparksys-lo