云原生架构下的业务日志收集[1]

于 2021-09-14 19:00:00 发布
阅读量356 收藏 1
点赞数
文章标签： 云原生 架构
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/qq_32106893/article/details/120300232
版权

468020d6b13cdfcdf37e4ec9984c619b.gif

ELK日志收集（filebeat-kafka-logstash-elasticsearch-kibana），在常规非容器模式下，可以通过filebeat采集日志目录，匹配正则规则，来收集日志。而容器化的日志收集比起常规收集容易出现更多的问题。

问题一：以docker容器为例，日志通常是在控制台进行标准输出，无固定的日志文件，收集难度较大。

问题二：docker的日志输出，通常会映射输出到虚拟机的**/var/lib/docker/containers/*/*.log**目录，目录内容较为繁琐手动收集比较困难

问题三：相对于非容器模式下，容器化k8s,docker swram集群管理更侧重于对资源的管理和调度，应用在实际部署中是无序且随时变化的，采集日志难度较大。

ec3829eecd827565ddcfa444bbb11d8a.png

目前对k8s集群日志的采集方案有两种，大家可根据实际情况具体选择：

方式一：daemonset 方式采集日志

实现技术原理：

日志收集的filebeat服务进程需要在每个node节点上运行，符合kuberntes daemonset的实现模式。通常来说日志收集代理，网络插件代理都会采用这种方式部署。

具体部署方式：

[root@172 elk]# cat filebeat-k8s-kafka-7.6.yml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: hub-sh.aijidou.com/library/filebeat:7.6.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          securityContext:
            runAsUser: 0
          resources:
            limits:
              cpu: 3000m
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
            - name: inputs
              mountPath: /usr/share/filebeat/inputs.d
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: inputs
          configMap:
            defaultMode: 0600
            name: filebeat-inputs
        - name: data
          hostPath:
            path: /var/lib/filebeat7.6-data
            type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
<一>日志进行通配符配置
[root@172 elk]# cat filebeat-configmap.yml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: true
          templates:
            - condition:
                equals:
                  kubernetes.namespace: kong
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline:
                    pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}'
                    negate: true
                    match: after
                  ignore_older: 168h
    processors: 
      - decode_json_fields:
          # fields_under_root: true
          fields: ['message'] 
          target: ""
          overwrite_keys: true
          process_array: true
          max_depth: 1
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch:
      hosts: [ "http://172.16.XX.XXX:9200" ]
      username: "elastic"
      password: "XXXXXXXXXXX"
    output.kafka:
      hosts: ["172.16.20.x1:9092","172.16.20.x2:9092","172.16.20.x3:9092"]
      topics:
        - topic: "kong-proxy-log"
          when.contains:
            kubernetes.container.name: "proxy"
            kubernetes.namespace: "kong"
<二>日志进行逐一匹配

[root@172 elk]# cat filebeat-configmap.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*xxx*parkingcp-*.log
      multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}'
      multiline.negate: true
      multiline.match: after
      ignore_older: 168h
      exclude_lines: ['^DBG']
      fields.topic: xxx-parkingcp
    #可作为公共配置存在 
    processors: 
      - decode_json_fields:
          fields: ['message'] 
          target: ""
          overwrite_keys: true
          process_array: true
          max_depth: 1
      - add_kubernetes_metadata:  
          host: ${NODE_NAME}  
          matchers:  
          - logs_path:  
              logs_path: "/var/log/containers/"  
      - drop_fields:
          fields: ["agent.ephemeral_id","agent.hostname","agent.id","agent.type","agent.version","ecs.version","host.name","input.type","kubernetes.container.image","kubernetes.node.name","kubernetes.pod.labels.app","kubernetes.pod.labels.pod-template-hash","kubernetes.statefulset.name","kubernetes.pod.uid","kubernetes.replicaset.name","log.file.path","log.offset"] 
    output.kafka:
      hosts: ["172.16.xx.66:9092","172.16.xx.67:9092","172.16.xx.68:9092"]
      topics:
        - topic: xxx-parkingcp-log
          when.contains:
            fields.topic: xxx-parkingcp

方式二：sidecar 方式采集日志

实现技术原理：

sidecar模式是Kubernetes引入的设计模式，指的是在同一Pod中，除了承载主要业务逻辑的容器外，还运行一个称为sidecar的辅助容器，提供一些通用功能支持。

使用Sidebar模式的filebeat容器处理日志，可以让应用不用关心日志发送到哪里，仅输出到stdout就可以。容器的输出会被同一Pod中的SideCar辅助filebeat容器截取，并将日志收集，并output。

具体部署方式：

以部署java spring boot项目为例，
[root@demo01 sidecar]#ls
Dockerfile
configmap.yaml
javademo.yaml
run.sh
[root@demo01 sidecar]# cat Dockerfile
FROM centos:7
RUN mkdir /usr/local/app/logs -p
COPY *.jar /usr/local/app/app.jar
COPY run.sh /usr/local/app/run.sh
COPY LAST_COMMIT.log /usr/local/app/
WORKDIR /usr/local/app/
EXPOSE 8080:8080
#ENTRYPOINT java -Xmx1000m -Xms1000m -Dspring.profiles.active=$profiles -jar app.jar >./logs/output.log 2>&1 &
#ENTRYPOINT ["sh","-c","nohup java -Xmx3000m -Xms3000m -Dspring.profiles.active=$profiles -jar app.jar >./logs/output.log 2>&1 &"]
ENTRYPOINT ["sh","-c","sh run.sh"]
#ENTRYPOINT ["tail","-f","/usr/local/app/logs/output.log"]
[root@demo01 sidecar]# cat configmap.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: xxx
data:
  filebeat.yml: |
    filebeat.inputs:
      - type: log
        enable: true
        paths:
        - /log/output.log
        multiline.pattern: \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}
        multiline.negate: true
        multiline.match: after
        exclude_lines: ['^DBG','WARN']
        fields:
          type_name: '${type_name}'#索引特定配置
          pod_name: '${pod_name}'
          pod_ip: '${pod_ip}'
        ignore_older: 168h
    output.kafka: 
      hosts: ["172.16.20.x1:9092","172.16.20.x2:9092","172.16.20.x3:9092"]        #指定输出数据到kafka集群上，地址与端口号想对应
      topic: '%{[fields][type_name]}'    #指定要发送数据到kafka集群的哪个topic，与上述的"fields: type_name："相对应
      partition.round_robin:         #开启kafka的partition分区
        reachable_only: true
      compression: gzip
      required_acks: 1
      max_message_bytes: 10000000    #压缩格式字节大小
[root@demo01 sidecar]# cat javademo.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: xxxx
  name: javademo-deployment
  labels:
    app: javademo-deployment
spec:
  selector:
    matchLabels:
      app: javademo-deployment
  replicas: 2
  minReadySeconds: 40
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: javademo-deployment
    spec:
      containers:
      - image: elastic/filebeat:7.6.0
        name: filebeat
        command: ["filebeat"]
        args: ["-c","/opt/filebeat/filebeat.yml","-e", "-strict.perms=false"]
        env:
        - name: pod_ip
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: pod_name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: type_name
          value: "xxxx-javademo"
        resources:
          limits:
            cpu: 2000m
            memory: 1Gi
        volumeMounts:
        - name: outputlog
          mountPath: /log
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: filebeat-config
          mountPath: /opt/filebeat
      - name: javademo
        image:  javademo:TARGET_VERSION
        ports:
        - containerPort: 8080
        env:
          - name: profiles
            value: "test"
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 300m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /javademo/ok
            port: 8080
          initialDelaySeconds: 35
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /javademo/ok
            port: 8080
          initialDelaySeconds: 110
          periodSeconds: 35
        volumeMounts:
          - name: outputlog
            mountPath: /usr/local/app/logs/
      volumes:
        - name: outputlog
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config
            items:
            - key: filebeat.yml
              path: filebeat.yml
        - name: data
          emptyDir: {}
      imagePullSecrets:
        - name: default-secret
      restartPolicy: Always
[root@demo01 sidecar]# cat run.sh
#!/bin/sh
export JAVA_HOME=/usr/local/jdk1.8.0_191
PATH=$PATH:$JAVA_HOME/bin
java -Dspring.profiles.active=$profiles -jar app.jar > ./logs/output.log 2>&1 &
tail -f /usr/local/app/logs/output.log
[root@test001 filebeat]# docker build -t javademo:1.0.0 .
[root@test001 filebeat]# docker images |grep filebeat
elastic/filebeat                                                        7.6.0                         cbd2cf26beba        19 months ago       364MB
javademo
                                                                1.0.0                         cbd2cf26beba        19 months ago       364MB
[root@test001 filebeat]# kubectl apply -f configmap.yaml
[root@test001 filebeat]# kubectl apply -f javademo.yaml

对比几种日志收集方式


daemonset 方式采集日志（通配符配置）daemonset 方式采集日志（逐一模块配置）sidecar方式日志收集
优点
安装配置最简单，便于快速配置收集可针对单个模块特性收集匹配，非常灵活；资源消耗少；对应用无侵入

可针对单个模块特性收集匹配可针对不同级别的日志进行分类别收集（ERROR,WARN,INFO,DEBUG）；

低耦合；

缺点

无法对具体模块的收集和

过滤日志规则进行配置

配置收集规则较为复杂；

配置收集规则较为复杂；

资源消耗成本较高；

采集方式
收集容器标准输出日志收集容器标准输出日志收集容器标准输出日志或者收集自定义日志目录

00e677ce5ae8789b4bfdb185e7228f0b.gif

精选原创推荐

docker容器部署Spring Profile参数策略

linux云服务器存储分区热挂载

Node的进程管理工具pm2

拒绝夜间上线：Eureka热部署服务

大厂必备：ansible 自动化工具

亲测好评：国产开源APM运维监控

APM钉钉告警二次开发pinpoint（附带源码）

Prometheus + Granafa 构建高大上的MySQL监控平台

syncd一款高效开源的代码部署工具

高效开发:IntelliJIDEA的这些Debug技巧你都知道吗

运维必备：goreplay流量复制工具

一款SQL高效审计工具|解放DBA双手

实用技能：git子模块功能

评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值