MINIO多节点多驱动部署+nginx负载均衡

本文链接：https://blog.youkuaiyun.com/weixin_46258856/article/details/134643388

MinIO 是一种高性能、S3 兼容的对象存储。它专为大规模 AI/ML、数据湖和数据库工作负载而构建。为了搭建项目文件数据的存储平台，本文记录了在一个ubuntu20.4内网服务器中（可以使用apt）进行MINIO多节点多驱动部署的过程。
以下所有步骤参考 MINIO官方文档：

一、预准备：

1.网络和防火墙

默认MINIO 服务器API端口9010（汇集到9000），静态MinIO 控制台端口9001（汇集到443）

sudo ufw allow 9000
sudo ufw allow 9010
sudo ufw allow 9001
sudo ufw allow 443

2.设置连续主机名（DNS或host）

sudo vim /etc/hosts

10.80.68.229 minio.example.com
10.80.68.229 minio-01.example.com
10.80.68.230 minio-02.example.com
10.80.68.231 minio-03.example.com
10.80.68.232 minio-04.example.com

3.具有顺序安装的本地 JBOD 存储（这里IT已经配好了）

sudo vim /etc/fstab

# /etc/fstab: static file system information.

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/disk/by-uuid/*** /data1 xfs defaults 0 1
/dev/disk/by-uuid/*** /data2 xfs defaults 0 1
/dev/disk/by-uuid/*** /data3 xfs defaults 0 1
/dev/disk/by-uuid/*** /data4 xfs defaults 0 1

4.时间同步（已配好）

systemctl status systemd-timesyncd

二、部署MINIO:

1.在每个节点上安装 MinIO 二进制文件

wget <https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20231101183725.0.0_amd64.deb> -O minio.deb
sudo dpkg -i minio.deb

2.配置systemd服务文件

RPM 和 DEB 包会自动将 MinIO 安装到必要的系统路径并为systemctl创建一个minio服务（二进制文件包需要手动创建）

此处修改你的用户名（User）和组（Group），其他可不变：

sudo vim /usr/lib/systemd/system/minio.service


[Unit]
Description=MinIO
Documentation=<https://min.io/docs/minio/linux/index.html>
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio02
Group=minio02
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (<https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)>
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

3.为你的用户配置组并开通权限

（可选）如果需要可以添加组和用户：

groupadd -r minio-user
useradd -M -r -g minio-user minio-user

（必须）为现有用户和组开通权限（:）：

chown minio02:minio02 /data1 /data2 /data3 /data4

4.创建服务环境文件

创建环境文件并编辑：

sudo vim /etc/default/minio


# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)

MINIO_VOLUMES="http://minio-0{1...4}.example.com:9000/data{1...4}"

# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001 --address :9010"


# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=minioadmin

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=minio-secret-key-CHANGE-ME

# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="https://minio.example.com:9000"

5.配置nginx负载均衡（仅在主节点）

安装 nginx

sudo apt update
sudo apt install nginx

修改 nginx.conf

upstream minio_server {
    least_conn;
    server minio-01:9010;
    server minio-02:9010;
    server minio-03:9010;
    server minio-04:9010;
}

upstream minio_console {
    ip_hash;
    server minio-01:9001;
    server minio-02:9001;
    server minio-03:9001;
    server minio-04:9001;
}


server {
    listen       9000 ssl;
    server_name  minio-dev.example.com;
  
    include /etc/nginx/conf.d/nginx.header;
  
    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    proxy_request_buffering off;

    proxy_buffer_size 16k;
    proxy_buffers 16 4k;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio_server;
    }
}

server {
    listen       443 ssl;
    server_name  minio-dev.example.com;

    include /etc/nginx/conf.d/nginx.header;

    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    proxy_request_buffering off;

    proxy_buffer_size 16k;
    proxy_buffers 16 4k;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
        
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        chunked_transfer_encoding off;

        proxy_pass http://minio_console;
    }
}

6.（四个节点同时）启动MINIO

sudo systemctl start minio.service

可以用以下命令查看日志与报错：

sudo systemctl status minio.service
journalctl -f -u minio.service

设置开机自启动：

sudo systemctl enable minio.service

7.打开控制台

https://minio-dev.example.com

初始管理员密码为环境变量minio中设置的内容：
在这里插入图片描述
另：

重载命令

sudo systemctl daemon-reload
sudo systemctl restart minio.service

三、遇到的问题

问题1：可以打开9001端口但是无法打开9000端口

原因：定位到是nginx配置不成功，打开error.log文件发现是端口被占用

解决：发现是nginx中upstream处的端口和server中的端口重复了，修改端口后成功打开界面

问题2：9000端口无法加密

原因：MINIO_OPTS一开始只指明了控制台端口，未指明服务端口，在nginx服务下服务端口默认指向了9000而不是9010，
占用了9000端口导致无法加密

解决： 引导minio服务流向9010端口，MINIO_OPTS="--console-address :9001 --address :9010"