kubernetes下用logstash代替efk中的fluentd

前言

目前我们的系统架构，从阿里云的docker compose(swarm)迁移到了自建的kubernetes,而我们的日志系统也几经周折。从最开始的阿里云docker compose(swarm)环境下的elk + kafka，使用了一段时间后由于老板觉得阿里云的kafka收费价格不怎么划算，并且线上服务器的资源吃紧，然后切换到了阿里云的日志服务。线上切换到kubernetes以后，又使用efk搜集了一段时间日志，后来发现fluentd搜集的日志存在一些延迟问题和准确性问题，并且变更配置搜集特定的日志时较为繁琐，于是准备把fluentd换为logstash + rabbit,期间对比了下redis,rabbit和kafka，最终选择了在用的自建的rabbit集群，由于近期事情较多，精力有限，此文只会阐述关于logstash和rabbit相关使用的过程，关于rabbit自建集群，以及k8s的efk，以及elk搭建，后面会逐一补充。

简要步骤

1. 编写logstash的配置文件logstash.conf 建议先在本地使用docker部署logstash，然后在本地进行测试，成功后再部署到kubernetes环境

input {
    rabbitmq {
      type => "app-log"
      host => "your rabbitmq host"
      user => "your user"
      password => "your password"
      port => 5672
      exchange => "exchange_logs"
      exchange_type => "direct"
      queue => "queue_logs"
      ack => false
      durable => true
    }
   }

   output {
    elasticsearch {
      hosts => ["elasticsearch.kube-ops:9200"]
      index => "logstash-%{type}-%{+YYYY.MM.dd}"
      document_type => "%{type}"
      flush_size => 20000
      idle_flush_time => 10
      template_overwrite => true
    }
    stdout {
        codec => rubydebug
    }
   }

output同时输出到es和控制台，便于调试
2. 编写logstash的k8s configMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: logstash-conf
  namespace: kube-ops
data:
  logstash.conf: |
   input {
    rabbitmq {
      type => "app-log"
      host => "rabbitmq...."
      user => "..."
      password => "..."
      port => 5672
      exchange => "exchange_logs"
      exchange_type => "direct"
      queue => "queue_logs"
      ack => false
      durable => true
    }
   }

   output {
    elasticsearch {
      hosts => ["elasticsearch.kube-ops:9200"]
      index => "logstash-%{type}-%{+YYYY.MM.dd}"
      document_type => "%{type}"
      flush_size => 20000
      idle_flush_time => 10
      template_overwrite => true
    }
    stdout {
        codec => rubydebug
    }
   }

执行

kubectl create -f logstash-configMap.yaml

1. 建议在本地先使用docker进行调试，如果没有问题，则可以进行下一步，部署到k8s中

docker run -d  -v ~/a/docker/efk-deploy/1.7/bac/es2/qatrans/logstash/logstash.conf:/etc/logstash/logstash.conf  logstash:5.6.4 logstash -f /e
tc/logstash/logstash.conf

1. 部署到k8s
   logstash.yaml如下：

apiVersion: v1
kind: Service
metadata:
  name: logstash
  namespace: kube-ops
spec:
  ports:
  - port: 5044
    targetPort: beats
  selector:
    type: logstash
  clusterIP: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: logstash
  namespace: kube-ops
spec:
  template:
    metadata:
      labels:
        type: logstash
    spec:
      containers:
      - image: anteoy/logstash:5.6.4
        name: logstash
        ports:
        - containerPort: 5044
          name: beats
        command:
        - logstash
        - '-f'
        - '/etc/logstash_c/logstash.conf'
        volumeMounts:
        - name: config-volume
          mountPath: /etc/logstash_c/
        resources:
          limits:
            cpu: 1000m
            memory: 2048Mi
          # requests:
          #   cpu: 512m
          #   memory: 512Mi
      volumes:
      - name: config-volume
        configMap:
          name: logstash-conf
          items:
          - key: logstash.conf
            path: logstash.conf

执行：

$ kubectl create -f logstash.yaml

参考

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html
https://gist.github.com/dblessing/a9d5a68da56eb451553a