kubernetes部署rocketMQ集群，(ns-master-2slave)*2

RocketMQ介绍

Apache RocketMQ是一个分布式消息传递和流媒体平台，具有低延迟、高性能和可靠性、万亿级别的容量和灵活的可伸缩性。

消息队列作为高并发系统的核心组件之一，能够帮助业务系统解构提升开发效率和系统稳定性。主要具有以下优势：

削峰填谷（主要解决瞬时写压力大于应用服务能力导致消息丢失、系统奔溃等问题）
系统解耦（解决不同重要程度、不同能力级别系统之间依赖导致一死全死）
提升性能（当存在一对多调用时，可以发一条消息给消息系统，让消息系统通知相关系统）
蓄流压测（线上有些链路不好压测，可以通过堆积一定量消息再放开来压测）

目前主流的MQ主要是Rocketmq、kafka、Rabbitmq，Rocketmq相比于Rabbitmq、kafka具有主要优势特性有：

• 支持事务型消息（消息发送和DB操作保持两方的最终一致性，rabbitmq和kafka不支持）
• 支持结合rocketmq的多个系统之间数据最终一致性（多方事务，二方事务是前提）
• 支持18个级别的延迟消息（rabbitmq和kafka不支持）
• 支持指定次数和时间间隔的失败消息重发（kafka不支持，rabbitmq需要手动确认）
• 支持consumer端tag过滤，减少不必要的网络传输（rabbitmq和kafka不支持）
• 支持重复消费（rabbitmq不支持，kafka支持）

同步刷盘、异步刷盘

RocketMQ的消息是存储到磁盘上的，这样既能保证断电后恢复，又可以让存储的消息量超出内存的限制。

RocketMQ为了提高性能，会尽可能地保证磁盘的顺序写。消息在通过Producer写入RocketMQ的时候，有两种

写磁盘方式：

  1）异步刷盘方式：在返回写成功状态时，消息可能只是被写入了内存的PAGECACHE，写操作的返回快，吞吐量大；当内存里的消息量积累到一定程度时，统一触发写磁盘操作，快速写入 
    优点：性能高
    缺点：Master宕机，磁盘损坏的情况下，会丢失少量的消息, 导致MQ的消息状态和生产者/消费者的消息状态不一致
　2）同步刷盘方式：在返回应用写成功状态前，消息已经被写入磁盘。具体流程是，消息写入内存的PAGECACHE后，立刻通知刷盘线程刷盘，然后等待刷盘完成，刷盘线程执行完成后唤醒等待的线程，给应用返回消息写成功的状态。     
    优点：可以保持MQ的消息状态和生产者/消费者的消息状态一致
    缺点：性能比异步的低

同步刷盘还是异步刷盘，是通过Broker配置文件里的flushDiskType参数设置的，这个参数被设置成SYNC_FLUSH, ASYNC_FLUSH中的一个

同步复制、异步复制

如果一个broker组有Master和Slave，消息需要从Master复制到Slave上，有同步和异步两种复制方式。

   1）同步复制方式：等Master和Slave均写成功后才反馈给客户端写成功状态
        优点：如果Master出故障，Slave上有全部的备份数据，容易恢复，消费者仍可以从Slave消费, 消息不丢失
        缺点：增大数据写入延迟，降低系统吞吐量，性能比异步复制模式略低，大约低10%左右，发送单个Master的响应时间会略高

   2）异步复制方式：只要Master写成功即可反馈给客户端写成功状态
        优点：系统拥有较低的延迟和较高的吞吐量. Master宕机之后，消费者仍可以从Slave消费，此过程对应用透明，不需要人工干预，性能同多个Master模式几乎一样
        缺点：如果Master出了故障，有些数据因为没有被写入Slave，而丢失少量消息。

同步复制和异步复制是通过Broker配置文件里的brokerRole参数进行设置的，这个参数可以被设置成ASYNC_MASTER、SYNC_MASTER、SLAVE三个值中的一个。
　　实际应用中，推荐把Master和Slave设置成ASYNC_FLUSH的异步刷盘方式，主从之间配置成SYNC_MASTER的同步复制方式，这样即使有一台机器出故障，仍然可以保证数据不丢。

部署

本例为K8S上部署，基于最新版的rocketMQ4.8.0制作的镜像，采用Statefulset模式，单个Pod部署一个name-server，一个brocker-master和两个brocker-slave，通过configmap管理配置文件。
先启动service，因为服务都在K8S集群调用，所有用了clusterIP，同时固定了clusterIP地址

apiVersion: v1
kind: Service
metadata:
  name: rocketmq1
  labels:
    app: rocketmq1
spec:
  type: ClusterIP
#指定IP地址
  clusterIP: 10.111.64.218
  ports:
    - port: 9876
      targetPort: 9876
      name: rmqns
    - port: 10911
      targetPort: 10911
      name: rmqbroker
    - port: 10921
      targetPort: 10921
      name: rmqbrokerslave
    - port: 10931
      targetPort: 10931
      name: rmqbrokerslave2
  selector:
    app: rocketmq1
  
    ---
apiVersion: v1
kind: Service
metadata:
  name: rocketmq2
  labels:
    app: rocketmq2
spec:
  type: ClusterIP
#指定IP地址
  clusterIP: 10.111.64.219
  ports:
    - port: 9876
      targetPort: 9876
      name: rmqns2
    - port: 10961
      targetPort: 10961
      name: rmqbroker2
    - port: 10971
      targetPort: 10971
      name: rmqbrokerslave2
    - port: 10981
      targetPort: 10981
      name: rmqbrokerslave22
  selector:
    app: rocketmq2   

通过命令行创建configmap，将配置文件由K8S管理,配置文件基本一致，具体下载地址链接
https://github.com/pilgrimaz/k8s/tree/master/rocketmq/conf

kubectl create configmap rocket-config --from-file=./broker-a.conf --from-file=./broker-a-s.conf --from-file=./broker-a-s2.conf --from-file=./broker-b.conf --from-file=./broker-b-s.conf --from-file=./broker-b-s2.conf

创建Pod，此处采用statfulset模式部署，用的是hostPath挂载硬盘，建议使用nfs或者pvc挂载，便于重新调度后也保持数据正常
创建rocketmq1

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq1
  labels:
    app: rocketmq1

spec:
  serviceName: rocketmq1
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq1
  template:
    metadata:
      labels:
        app: rocketmq1
    spec:
      containers:
        - name: rocketmq
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT
              value: -Duser.home=/home/rocketmq
            - name: JAVA_OPT_EXT
              value:  -Xmn256m
          command: ["sh","mqnamesrv"]
          ports:
            - containerPort: 9876
          volumeMounts:
            - name: vol-logs
              mountPath: /home/rocketmq/logs
            - name: vol-store
              mountPath: /home/rocketmq/store
        - name: rmqbroker
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 440m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker",-c,"/etc/rocketmq/broker-a.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10909
            - containerPort: 10911
            - containerPort: 10912
          volumeMounts:
            - name: vol-logs-a
              mountPath: /home/rocketmq/logs
            - name: vol-store-a
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
        - name: rmqbroker-slave
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker","-c","/etc/rocketmq/broker-a-s.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10919
            - containerPort: 10921
            - containerPort: 10922
          volumeMounts:
            - name: vol-logs-as
              mountPath: /home/rocketmq/logs
            - name: vol-store-as
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
        - name: rmqbroker-slave2
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker","-c","/etc/rocketmq/broker-a-s2.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10929
            - containerPort: 10931
            - containerPort: 10932
          volumeMounts:
            - name: vol-logs-as2
              mountPath: /home/rocketmq/logs
            - name: vol-store-as2
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
      volumes:
        - name: vol-logs
          hostPath:
            path: /data1/rocketmq/logs-1
            type: DirectoryOrCreate
        - name: vol-store
          hostPath:
            path: /data1/rocketmq/store-1
            type: DirectoryOrCreate
        - name: vol-conf
          configMap:
            name: rocketmq-config
        - name: vol-logs-a
          hostPath:
            path: /data1/rocketmq/logs-1a
            type: DirectoryOrCreate
        - name: vol-store-a
          hostPath:
            path: /data1/rocketmq/store-1a
            type: DirectoryOrCreate
        - name: vol-logs-as
          hostPath:
            path: /data1/rocketmq/logs-1as
            type: DirectoryOrCreate
        - name: vol-store-as
          hostPath:
            path: /data1/rocketmq/store-1as
            type: DirectoryOrCreate
        - name: vol-logs-as2
          hostPath:
            path: /data1/rocketmq/logs-1as2
            type: DirectoryOrCreate
        - name: vol-store-as2
          hostPath:
            path: /data1/rocketmq/store-1as2
            type: DirectoryOrCreate

创建rocketmq2

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rocketmq2
  labels:
    app: rocketmq2

spec:
  serviceName: rocketmq2
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq2
  template:
    metadata:
      labels:
        app: rocketmq2
    spec:
      containers:
        - name: rocketmq
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT
              value: -Duser.home=/home/rocketmq
            - name: JAVA_OPT_EXT
              value:  -Xmn256m
          command: ["sh","mqnamesrv"]
          ports:
            - containerPort: 9876
          volumeMounts:
            - name: vol-logs
              mountPath: /home/rocketmq/logs
            - name: vol-store
              mountPath: /home/rocketmq/store
        - name: rmqbroker
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 440m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker","-c","/etc/rocketmq/broker-b.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10959
            - containerPort: 10961
            - containerPort: 10962
          volumeMounts:
            - name: vol-logs-a
              mountPath: /home/rocketmq/logs
            - name: vol-store-a
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
        - name: rmqbroker-slave
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker","-c","/etc/rocketmq/broker-b-s.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10969
            - containerPort: 10971
            - containerPort: 10972
          volumeMounts:
            - name: vol-logs-as
              mountPath: /home/rocketmq/logs
            - name: vol-store-as
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
        - name: rmqbroker-slave2
          image: pilgrima/rocketmq:broker-4.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 450m
              memory: 2000Mi
            requests:
              cpu: 400m
              memory: 2000Mi
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_OPT_EXT
              value: -Duser.home=/home/rocketmq -Xmx2048M -Xms1024M -Xmn256m
          command: [ "sh","mqbroker","-c","/etc/rocketmq/broker-b-s2.conf","autoCreateTopicEnable=true" ]
          ports:
            - containerPort: 10979
            - containerPort: 10981
            - containerPort: 10982
          volumeMounts:
            - name: vol-logs-as2
              mountPath: /home/rocketmq/logs
            - name: vol-store-as2
              mountPath: /home/rocketmq/store
            - name: vol-conf
              mountPath: /etc/rocketmq/
      volumes:
        - name: vol-logs
          hostPath:
            path: /data1/rocketmq/logs-2
            type: DirectoryOrCreate
        - name: vol-store
          hostPath:
            path: /data1/rocketmq/store-2
            type: DirectoryOrCreate
        - name: vol-conf
          configMap:
            name: rocketmq-config
        - name: vol-logs-a
          hostPath:
            path: /data1/rocketmq/logs-2b
            type: DirectoryOrCreate
        - name: vol-store-a
          hostPath:
            path: /data1/rocketmq/store-2b
            type: DirectoryOrCreate
        - name: vol-logs-as
          hostPath:
            path: /data1/rocketmq/logs-2bs
            type: DirectoryOrCreate
        - name: vol-store-as
          hostPath:
            path: /data1/rocketmq/store-2bs
            type: DirectoryOrCreate
        - name: vol-logs-as2
          hostPath:
            path: /data1/rocketmq/logs-2bs2
            type: DirectoryOrCreate
        - name: vol-store-as2
          hostPath:
            path: /data1/rocketmq/store-2bs2
            type: DirectoryOrCreate

通过kubectl get pod查看两个Pod的启动情况
通过kubectl exec -it rockermq1 -c rmqbrocker -- sh mqadmin clusterList -n 172.0.0.1:9876查看集群状态

有条件的可以部署一个rocketmq-console监控集群状态，至此部署完毕

参考

https://blog.youkuaiyun.com/fenglailea/article/details/113116012