华为鲲鹏云ARM64 CentOS部署Mesos集群和Marathon

1. 在集群所有主机上编辑/etc/hosts
将集群内所有主机的IP和hostname添加进去。

2. 搭建一个zookeeper集群
这边使用的docker-compose跑的zk，分别在三个节点上各跑一个组成集群。docker-compose.yml（注意yml文件中的ZOO_MY_ID，三个节点分别是1，2，3）：

version: '2'
services:
  zookeeper:
    image: arm64v8/zookeeper:3.3
    environment:
      ZOO_SERVERS: server.1=campusphere-kp02:2888:3888 server.2=campusphere-kp03:2888:3888 server.3=campusphere-kp04:2888:3888
      ZOO_MY_ID: 1
      ZOO_PORT: 2182
    network_mode: "host"
    restart: always

使用docker-compose -f /opt/docker/docker-compose.yml up -d启动zk。

3. 集群中找一台机器用来编译mesos。
下载mesos源码包：http://mirrors.tuna.tsinghua.edu.cn/apache/mesos/1.8.1/mesos-1.8.1.tar.gz

4. 搭建jdk+maven环境
4.1. 下载jdk和maven。jdk到oracle官网下载， maven下载地址：http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz

4.2. 解压jdk和maven

tar -zxvf jdk-8u221-linux-arm64-vfp-hflt.tar.gz -C /usr/local
tar -zxvf apache-maven-3.6.1-bin.tar.gz -C /usr/local

4.3. 编辑/etc/profile，添加jdk和maven的环境变量

export MAVEN_HOME=/usr/local/apache-maven-3.6.1
export JAVA_HOME=/usr/local/jdk1.8.0_221
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

4.4. 使环境变量生效source /etc/profile，执行java -version和mvn -v检查环境是否正常。

5. 安装依赖

yum install -y tar wget git
yum install cppunit-devel
yum groupinstall -y "Development Tools"
yum install -y python-devel java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel

6.解压mesos源码包并编译(编译和安装时间比较长)

mkdir -p /usr/local/mesos
tar -zxvf mesos-1.8.1.tar.gz -C /usr/local/mesos
cd /usr/local/mesos/mesos-1.8.1
./config --prefix=/usr/local/mesos
make -j6
make -j6 install

7. 编辑/etc/profile
添加mesos的环境变量(集群内所有的主机都添加)

export MESOS_HOME=/usr/local/mesos
export PATH=${PATH}:${MESOS_HOME}/sbin:${MESOS_HOME}/bin

8. 使环境变量生效

source /etc/profile

9. 配置mesos

cd /usr/local/mesos/etc/mesos
cp mesos-agent-env.sh.template mesos-agent-env.sh
cp mesos-master-env.sh.template mesos-master-env.sh
cp mesos-deploy-env.sh.template mesos-deploy-env.sh
cp mesos-slave-env.sh.template mesos-slave-env.sh
chmod +x *.sh

9.1 修改mesos-master-env.sh

# This file contains environment variables that are passed to mesos-master.
# To get a description of all options run mesos-master --help; any option
# supported as a command-line option is also supported as an environment
# variable.

# Some options you're likely to want to set:
export MESOS_log_dir=/data/mesos/log
export MESOS_work_dir=/data/mesos/data
export MESOS_ZK=zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos
export MESOS_quorum=2

# 解释：
# MESOS_log_dir：log输出的目录，如果不设置，默认不会产生log
# MESOS_worker_dir:存放元数据的目录，如果对配置文件修改后，可能需要删除该目录下的文件，然后再重新启动。
# MESOS_ZK：Zookeeper的相关信息，我这边的zk配置的是域名，使用的bind搭建的域名服务器
# MESOS_quorum：用于HA，根据master的个数设定：有1个master，MESOS_quorum=1；有3个master，MESOS_quorum=2；有5个master，MESOS_quorum=3。

9.2 修改mesos-agent-env.sh和mesos-slave-env.sh（这两个文件一样）

# This file contains environment variables that are passed to mesos-agent.
# To get a description of all options run mesos-agent --help; any option
# supported as a command-line option is also supported as an environment
# variable.

# You must at least set MESOS_master.

# The mesos master URL to contact. Should be host:port for
# non-ZooKeeper based masters, otherwise a zk:// or file:// URL.
export MESOS_master=zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos

# Other options you're likely to want to set:
export MESOS_log_dir=/data/mesos/log
export MESOS_work_dir=/data/mesos/run
export MESOS_isolation=cgroups/cpu,cgroups/mem
export MESOS_containerizers=docker,mesos
export MESOS_switch_user=0
export MESOS_docker_config=/root/.docker/config.json

# 解释
# MESOS_isolation：隔离机制
# MESOS_containerizers：可用的容器实现机制，包括 mesos、external、docker
# MESOS_switch_user：是否用提交任务的用户身份来运行，默认为true
# MESOS_docker_config：用于仓库认证

10. 在编译mesos的机器上将编译好的mesos打包，并传到集群内的其他机器上解压

cd /usr/local
tar -zcvf mesos.tar.gz --exclude=mesos/mesos-1.8.1 mesos
scp mesos.tar.gz root@192.168.0.155:/usr/local/

11. 启动mesos应用

11.1 集群中的master节点上需要启动master

mesos-daemon.sh mesos-master

11.2 集群中的slave节点上启动agent

mesos-daemon.sh mesos-agent

11.3 本地使用master-ip:5050即可以打开mesos页面（先在本地把集群主机信息加入到hosts文件中）



12. 安装Marathon


12.1 下载marathon：http://downloads.mesosphere.com/marathon/v1.1.1/marathon-1.1.1.tgz


12.2 解压安装

tar -zxvf marathon-1.1.1.tgz -C /usr/local

12.3 修改启动脚本

cd /usr/local/marathon-1.1.1/bin
vi start
# 在脚本最前面声明MESOS_NATIVE_JAVA_LIBRARY，加上如下语句：
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/lib/libmesos.so

12.4 启动Marathon

nohup ./start --master zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/mesos --zk zk://zookeeper1.campusphere:2182,zookeeper2.campusphere:2182,zookeeper3.campusphere:2182/marathon --http_credentials wisedu:1qaz2wsx --event_subscriber http_callback &

12.5 本地使用ip:8080即可打开marathon页面，使用启动命令中的用户名和密码即可登录



13. 在Marathon上新建一个应用测试一下
点击“Create Application”，打开“JSON Mode”，将用于测试的应用的json贴进去，这边用了一个tomcat的镜像进行的测试：

{
  "id": "/tomcat-test",
  "cmd": null,
  "cpus": 0.2,
  "mem": 1024,
  "disk": 0,
  "instances": 2,
  "container": {
    "type": "DOCKER",
    "volumes": [],
    "docker": {
      "image": "campusphere-kp01:5000/tomcat-arm64v8:jdk8-corretto",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 0,
          "servicePort": 10001,
          "protocol": "tcp",
          "name": "tomcat",
          "labels": {}
        }
      ],
      "privileged": false,
      "parameters": [],
      "forcePullImage": true
    }
  },
  "healthChecks": [
    {
      "path": "/",
      "protocol": "HTTP",
      "portIndex": 0,
      "gracePeriodSeconds": 300,
      "intervalSeconds": 60,
      "timeoutSeconds": 20,
      "maxConsecutiveFailures": 3,
      "ignoreHttp1xx": false
    }
  ],
  "portDefinitions": [
    {
      "port": 10001,
      "protocol": "tcp",
      "labels": {}
    }
  ]
}

测试成功：