logstash部署(docker)与使用

友莘居士

已于 2024-12-19 11:05:33 修改

阅读量466

点赞数 4

文章标签： docker 容器云原生 big data

于 2024-12-17 19:18:59 首次发布

本文链接：https://blog.youkuaiyun.com/qq_37713191/article/details/144541986

版权

部署步骤

1.导入logstash镜像
docker load -i logstash7152.tar

2、创建挂载路径
mkdir -p /data/cloud/logstash/logs
mkdir -p /data/cloud/logstash/jdbc

3、设置挂载文件夹及里面的文件有读写权限
chmod -R 777 /data/cloud/logstash/logs/
chmod -R 777 /data/cloud/logstash/jdbc/

4、由于logstash的docker-compose无法自动创建挂在路径对应的内部路径，需要先运行镜像创建路径在对应挂载
docker run -d --name=logstash -p 5005:5000 elastic/logstash:7.15.2

5、复制docker部署的logstash内部配置文件到挂载路径
docker cp logstash:/usr/share/logstash/pipeline /data/cloud/logstash/
docker cp logstash:/usr/share/logstash/config /data/cloud/logstash/

6、关闭临时启动的logstash
docker stop logstash && docker rm logstash

7、编辑docker-compose-logstash.yml

version : '3'

networks:
  xx_cloud_network:
    external: true

services:
  logstash:
    image: elastic/logstash:7.15.2
    container_name: logstash
    volumes:
      - /data/cloud/logstash/pipeline:/usr/share/logstash/pipeline
      - /data/cloud/logstash/config:/usr/share/logstash/config
      - /data/cloud/logstash/logs:/usr/share/logstash/logs
      - /data/cloud/logstash/jdbc:/usr/share/logstash/jdbc
    ports:
      - 5000: 5000
    restart: always
    privileged: true
    networks:
      - xx_cloud_network

8、执行命令docker-compose -f docker-compose-logstash.yml up -d --build

9、配置文件说明

pipelines.yml：包涵多个logstash.conf

logstash.conf：具体的处理逻辑，可以写一个或多个，多个要在pipelines.yml同时配置

logstash.yml：主要参数为http.host指标REST端点的绑定地址，参考https://blog.youkuaiyun.com/weixin_33694620/article/details/88722734

10、使用实例

input {

    kafka {
        bootstrap_servers => "192.168.1.36:9092"
        #compression_type => "snappy"
        security_protocol => "SASL_PLAINTEXT"
        sasl_mechanism => "PLAIN"
        sasl_jaas_config => "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='kafkapwd';"
        topics => ["api_synchronize"]
        group_id=>"api_synchronize_group"
        codec => "json"
        type => "sys_api"
    }

      # syslog
    syslog {
        port => 5000
        add_field => {
        "source" => "syslog"
        }
    }

}

filter {

    mutate {
    # 删除字段
    remove_field => ["@version","type"]
    }

}


output {

    if [type] == "sys_api" {

        elasticsearch {
            hosts => "http://192.168.2.37:9100"
            user => "elastic"
            password => "espassword"
            index => "all-securityalarm-%{+YYYY}"
            manage_template => false
            ssl => false
        }

    }

    stdout {
        codec => "json"
    }
}