Openshift下使用RocketMQ operator部署Rocketmq并使用Prometheus + Grafana 监控

1.安装RocketMQ operator

2.新建project

oc new-project rocket-middleware

3. 新建ConfigMap

此为broker的配置文件

kind: ConfigMap
apiVersion: v1
metadata:
  name: broker-config
  namespace: rocket-middleware
data:
  BROKER_MEM: ' -Xms2g -Xmx2g -Xmn1g '
  broker-common.conf: |
    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
    brokerRole=ASYNC_MASTER

4.新建NameService

注意更换 storageClassName: xxxx

apiVersion: rocketmq.apache.org/v1alpha1
kind: NameService
metadata:
  name: name-service
  namespace: rocket-middleware
spec:
  size: 3
  resources:
    limits:
      cpu: 2000m
      memory: 4096Mi
    requests:
      cpu: 250m
      memory: 512Mi
  hostNetwork: true
  hostPath: /data/rocketmq/nameserver
  nameServiceImage: 'docker.io/apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0'
  imagePullPolicy: Always
  volumeClaimTemplates:
    - metadata:
        name: namesrv-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: xxxx
  dnsPolicy: ClusterFirstWithHostNet
  storageMode: StorageClass

---
kind: Service
apiVersion: v1
metadata:
  name: name-service-0
  namespace: rocket-middleware
spec:
  ports:
    - name: nameserver
      protocol: TCP
      port: 9876
      targetPort: 9876
    - name: master
      protocol: TCP
      port: 10911
      targetPort: 10911
    - name: slave
      protocol: TCP
      port: 11911
      targetPort: 11911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: name-service-0
---
kind: Service
apiVersion: v1
metadata:
  name: name-service-1
  namespace: rocket-middleware
spec:
  ports:
    - name: nameserver
      protocol: TCP
      port: 9876
      targetPort: 9876
    - name: master
      protocol: TCP
      port: 10911
      targetPort: 10911
    - name: slave
      protocol: TCP
      port: 11911
      targetPort: 11911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: name-service-1

---
kind: Service
apiVersion: v1
metadata:
  name: name-service-2
  namespace: rocket-middleware
spec:
  ports:
    - name: nameserver
      protocol: TCP
      port: 9876
      targetPort: 9876
    - name: master
      protocol: TCP
      port: 10911
      targetPort: 10911
    - name: slave
      protocol: TCP
      port: 11911
      targetPort: 11911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: name-service-2

5.新建Broker

apiVersion: rocketmq.apache.org/v1alpha1
kind: Broker
metadata:
  name: broker
  namespace: rocket-middleware
spec:
  size: 3
  brokerImage: 'docker.io/apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0'
  nameServers: 'name-service-0:9876;name-service-1:9876;name-service-2:9876;'
  resources:
    limits:
      cpu: 4000m
      memory: 4Gi
    requests:
      cpu: 1000m
      memory: 4Gi
  env:
    - name: BROKER_MEM
      valueFrom:
        configMapKeyRef:
          key: BROKER_MEM
          name: broker-config
  hostPath: /data/rocketmq/broker
  volumeClaimTemplates:
    - metadata:
        name: broker-storage
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: xxxx
  volumes:
    - configMap:
        items:
          - key: broker-common.conf
            path: broker-common.conf
        name: broker-config
      name: broker-config
  replicaPerGroup: 0
  storageMode: StorageClass

---
kind: Service
apiVersion: v1
metadata:
  name: broker-0-master-0
  namespace: rocket-middleware
spec:
  ports:
    - name: tcp
      protocol: TCP
      port: 10911
      targetPort: 10911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: broker-0-master-0

---
kind: Service
apiVersion: v1
metadata:
  name: broker-1-master-0
  namespace: rocket-middleware
spec:
  ports:
    - name: tcp
      protocol: TCP
      port: 10911
      targetPort: 10911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: broker-1-master-0

---
kind: Service
apiVersion: v1
metadata:
  name: broker-2-master-0
  namespace: rocket-middleware
spec:
  ports:
    - name: tcp
      protocol: TCP
      port: 10911
      targetPort: 10911
  type: ClusterIP
  selector:
    statefulset.kubernetes.io/pod-name: broker-2-master-0

6.新建Console

apiVersion: rocketmq.apache.org/v1alpha1
kind: Console
metadata:
  name: console
  namespace: rocket-middleware
spec:
  consoleDeployment:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: rocketmq-console
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: rocketmq-console
      template:
        metadata:
          labels:
            app: rocketmq-console
        spec:
          containers:
            - image: 'docker.io/apacherocketmq/rocketmq-console:2.0.0'
              name: console
              ports:
                - containerPort: 8080
                  protocol: TCP
  nameServers: 'name-service-0:9876;name-service-1:9876;name-service-2:9876;'

---
kind: Service
apiVersion: v1
metadata:
  name: rocketmq-console
  namespace: rocket-middleware
spec:
  ports:
    - name: console
      protocol: TCP
      port: 8080
      targetPort: 8080
  type: ClusterIP
  selector:
    app: rocketmq-console

---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: console
  namespace: rocket-middleware
spec:
  host: xxxxxxxxxxx
  to:
    kind: Service
    name: rocketmq-console
    weight: 100
  port:
    targetPort: console
  wildcardPolicy: None

7.暴露 RocketMQ 指标

Apache RocketMQ 需要一个 Exporter 来将其指标暴露为 Prometheus 格式。

1 部署 RocketMQ Exporter

kind: Deployment
apiVersion: apps/v1
metadata:
  name: rocketmq-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-exporter
  template:
    metadata:
      labels:
        app: rocketmq-exporter
    spec:
      containers:
        - resources:
            limits:
              cpu: 500m
              memory: 1Gi
            requests:
              cpu: 250m
              memory: 512Mi
          name: rocketmq-exporter
          command:
            - sh
            - '-c'
            - |
              java -jar rocketmq-exporter.jar --rocketmq.config.namesrvAddr='name-service-0:9876;name-service-1:9876;name-service-2:9876' --rocketmq.config.useTLS=false
          ports:
            - containerPort: 5557
              protocol: TCP
          image: 'apache/rocketmq-exporter:latest'

2. 创建 RocketMQ Exporter Service

apiVersion: v1
kind: Service
metadata:
  name: rocketmq-exporter
  labels:
    app: rocketmq-exporter
spec:
  ports:
    - name: metrics
      port: 5557
      targetPort: 5557
      protocol: TCP
  selector:
    app: rocketmq-exporter
  type: ClusterIP

部署以上资源：

oc apply -f rocketmq-exporter.yaml
oc apply -f rocketmq-exporter-service.yaml

8.配置 Prometheus 监控 RocketMQ

Prometheus 需要通过 ServiceMonitor 抓取 RocketMQ Exporter 的指标。

创建 ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    release: prometheus
  name: rocketmq-monitor
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: metrics
  namespaceSelector:
    matchNames:
      - rocket-middleware
  selector:
    matchLabels:
      app: rocketmq-exporter

部署 ServiceMonitor

oc apply -f rocketmq-servicemonitor.yaml

设置告警规则

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: rocketmq-rules
  namespace: gigp-rocketmq
spec:
  groups:
    - name: rocketmq-alerts
      rules:
        - alert: messageBacklog
          annotations:
            description: |
              消费组 {{ $labels.group}} 在 topic {{ $labels.topic}}的积压超过100。
              当前值：{{ $value}} , 请立即处理，否则可能导致消息堵死。
            summary: 'gigp rocketmq 消息挤压严重'
          for: 1m
          expr: |
            sum(rocketmq_producer_offset) by (topic) - on(topic) group_right sum(rocketmq_consumer_offset) by (group,topic) > 100
          labels:
            severity: critical
        - alert: messageBacklog
          annotations:
            description: |
              消费组 {{ $labels.group}} 在 topic {{ $labels.topic}}的积压超过10。
              当前值：{{ $value}} , 可能会出现消费堵塞或者异常，请及时排查。
            summary: 'gigp rocketmq 消费堆积过高'
          for: 1m
          expr: |
            sum(rocketmq_producer_offset) by (topic) - on(topic) group_right sum(rocketmq_consumer_offset) by (group,topic) > 10
          labels:
            severity: warning

9.在 Grafana 中配置仪表板

在 Grafana 中点击 + Create -> Import。

Dashboard ID: 14612

问题：RocketMQ-单条消息大小上限

单条消息大小上限 默认大小为4M

producer端会做校验，同时broker的MessageStoreConfig配置了也是4M，
客户端限制的消息大小，超过报错，同时服务端也会限制，所以需要跟服务端配合使用。

解决： 修改broker-config
加这么一句：maxMessageSize=104857600，修改为100MB

apiVersion: v1
kind: ConfigMap
metadata:
  name: broker-config
data:
  BROKER_MEM: ' -Xms2g -Xmx2g -Xmn1g '
  broker-common.conf: >
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    brokerRole=ASYNC_MASTER
    maxMessageSize=104857600
binaryData: {}

重启broker，读取最近的配置文件