【基础设施】【linux下在单机搭建MongoDB集群】【权限系统】

本文链接：https://blog.youkuaiyun.com/qq_43659174/article/details/88177223

跟redis相比，网上MongoDB的文章比较少，MongoDB集群的搭建大部分是多机器部署，为了测试方便，需要在本地机器部署MongoDB集群，参考了很多文章，终于搭建成功，记录下相关步骤。

需要注意的是，MongoDB更新很快，很多文章的语法过时，建议参考看官网文档，

环境版本：

mongodb 3.4.6

Centos 7.5

1、安装mongodb

#解压
tar -xzvf mongodb-linux-x86_64-3.4.6.tgz
#改名
mv mongodb-linux-x86_64-3.4.6 ../mongodb

#配置环境变量
vim /etc/profile
export MONGODB_HOME=/opt/mongodb
export PATH=$MONGODB_HOME/bin:$PATH
source /etc/profile

端口情况
config :21001,21002,21003
shard1:27011,27012,27013
shard2:27021,27022,27023
shard3:27031,27032,27033
mongos:20001,20002,20003
分别在每台机器建立conf、mongos、config、shard1、shard2、shard3目录

mkdir -p /opt/mongodb/conf
mkdir -p /opt/mongodb/config1/data
mkdir -p /opt/mongodb/config1/log
mkdir -p /opt/mongodb/config2/data
mkdir -p /opt/mongodb/config2/log
mkdir -p /opt/mongodb/config3/data
mkdir -p /opt/mongodb/config3/log
mkdir -p /opt/mongodb/mongos1/log
mkdir -p /opt/mongodb/mongos2/log
mkdir -p /opt/mongodb/mongos3/log
mkdir -p /opt/mongodb/shard1a/data
mkdir -p /opt/mongodb/shard1a/log
mkdir -p /opt/mongodb/shard1b/data
mkdir -p /opt/mongodb/shard1b/log
mkdir -p /opt/mongodb/shard1c/data
mkdir -p /opt/mongodb/shard1c/log
mkdir -p /opt/mongodb/shard2a/data
mkdir -p /opt/mongodb/shard2a/log
mkdir -p /opt/mongodb/shard2b/data
mkdir -p /opt/mongodb/shard2b/log
mkdir -p /opt/mongodb/shard2c/data
mkdir -p /opt/mongodb/shard2c/log
mkdir -p /opt/mongodb/shard3a/data
mkdir -p /opt/mongodb/shard3a/log
mkdir -p /opt/mongodb/shard3b/data
mkdir -p /opt/mongodb/shard3b/log
mkdir -p /opt/mongodb/shard3c/data
mkdir -p /opt/mongodb/shard3c/log

2、config server配置服务器

mongodb3.4以后要求配置服务器也创建副本集，不然集群搭建不成功。

添加配置文件

vi /opt/mongodb/conf/config1.conf
## 配置文件内容
systemLog:
    destination: file
    logAppend: true
    path: /opt/mongodb/config1/log/congigsrv.log
storage:
    dbPath: /opt/mongodb/config1/data
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: /opt/mongodb/config1/log/configsrv.pid
net:
    port: 21001
    bindIp: 0.0.0.0
replication:
    replSetName: configs
sharding:
    clusterRole: configsvr
## 配置文件end

vi /opt/mongodb/conf/config2.conf
## 配置文件内容
systemLog:
    destination: file
    logAppend: true
    path: /opt/mongodb/config2/log/congigsrv.log
storage:
    dbPath: /opt/mongodb/config2/data
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: /opt/mongodb/config2/log/configsrv.pid
net:
    port: 21002
    bindIp: 0.0.0.0
replication:
    replSetName: configs
sharding:
    clusterRole: configsvr
## 配置文件end

vi /opt/mongodb/conf/config3.conf
## 配置文件内容
systemLog:
    destination: file
    logAppend: true
    path: /opt/mongodb/config3/log/congigsrv.log
storage:
    dbPath: /opt/mongodb/config3/data
    journal:
        enabled: true
processManagement:
    fork: true
    pidFilePath: /opt/mongodb/config3/log/configsrv.pid
net:
    port: 21003
    bindIp: 0.0.0.0
replication:
    replSetName: configs
sharding:
    clusterRole: configsvr
## 配置文件end

启动三台服务器的config server

mongod -f /opt/mongodb/conf/config1.conf
mongod -f /opt/mongodb/conf/config2.conf
mongod -f /opt/mongodb/conf/config3.conf
登录任意一台配置服务器，初始化配置副本集

#连接
mongo --port 21001
#config变量
config = {
_id : "configs",
members : [
    {_id : 0, host : "127.0.0.1:21001" },
    {_id : 1, host : "127.0.0.1:21002" },
    {_id : 2, host : "127.0.0.1:21003" }
]
}

#初始化副本集
rs.initiate(config)
exit
其中，”_id” : “configs”应与配置文件中配置的 replicaction.replSetName 一致，”members” 中的 “host” 为三个节点的 ip 和 port
3、配置分片副本集(三台机器)

设置第一个分片副本集

vi /opt/mongodb/conf/shard1a.conf
#配置文件内容
systemLog:
    destination: file
    logAppend: true
    path: /opt/mongodb/shard1a/log/shard1.log
storage:
    dbPath: /opt/mongodb/shard1a/data
    journal:
        enabled: true
    wiredTiger:
        engineConfig:
            cacheSizeGB: 20
processManagement:
    fork: true
    pidFilePath: /opt/mongodb/shard1a/log/shard1.pid
net:
    port: 27011
    bindIp: 0.0.0.0
replication:
    replSetName: shard1
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard1b.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard1b/log/shard1.log
storage:
dbPath: /opt/mongodb/shard1b/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard1b/log/shard1.pid
net:
port: 27012
bindIp: 0.0.0.0
replication:
    replSetName: shard1
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard1c.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard1c/log/shard1.log
storage:
dbPath: /opt/mongodb/shard1c/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard1c/log/shard1.pid
net:
port: 27013
bindIp: 0.0.0.0
replication:
    replSetName: shard1
sharding:
    clusterRole: shardsvr
#配置文件end

启动三台服务器的shard1 server

mongod -f /opt/mongodb/conf/shard1a.conf
mongod -f /opt/mongodb/conf/shard1b.conf
mongod -f /opt/mongodb/conf/shard1c.conf
登陆任意一台服务器，初始化副本集

mongo --port 27011
#使用admin数据库
use admin
#定义副本集配置
config ={_id : "shard1",members : [
{_id : 0, host : "127.0.0.1:27011" },
{_id : 1, host : "127.0.0.1:27012" },
{_id : 2, host : "127.0.0.1:27013"}
]
}
#初始化副本集配置#
rs.initiate(config);

设置第二个分片副本集-------------------------------
vi /opt/mongodb/conf/shard2a.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard2a/log/shard1.log
storage:
dbPath: /opt/mongodb/shard2a/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard2a/log/shard1.pid
net:
port: 27021
bindIp: 0.0.0.0
replication:
    replSetName: shard2
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard2b.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard2b/log/shard1.log
storage:
dbPath: /opt/mongodb/shard2b/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard2b/log/shard1.pid
net:
port: 27022
bindIp: 0.0.0.0
replication:
    replSetName: shard2
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard2c.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard2c/log/shard1.log
storage:
dbPath: /opt/mongodb/shard2c/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard2c/log/shard1.pid
net:
port: 27023
bindIp: 0.0.0.0
replication:
    replSetName: shard2
sharding:
    clusterRole: shardsvr
#配置文件end

启动三台服务器的shard2 server

mongod -f /opt/mongodb/conf/shard2a.conf
mongod -f /opt/mongodb/conf/shard2b.conf
mongod -f /opt/mongodb/conf/shard2c.conf
登陆任意一台服务器，初始化副本集

mongo --port 27021
#使用admin数据库
use admin
#定义副本集配置
config ={_id : "shard2",members : [
{_id : 0, host : "127.0.0.1:27021" },
{_id : 1, host : "127.0.0.1:27022" },
{_id : 2, host : "127.0.0.1:27023"}
]
}
#初始化副本集配置
rs.initiate(config);
设置第二个分片副本集结束-------------------------------

设置第三个分片副本集-------------------------------
vi /opt/mongodb/conf/shard3a.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard3a/log/shard1.log
storage:
dbPath: /opt/mongodb/shard3a/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard3a/log/shard1.pid
net:
port: 27031
bindIp: 0.0.0.0
replication:
    replSetName: shard3
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard3b.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard3b/log/shard1.log
storage:
dbPath: /opt/mongodb/shard3b/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard3b/log/shard1.pid
net:
port: 27032
bindIp: 0.0.0.0
replication:
    replSetName: shard3
sharding:
    clusterRole: shardsvr
#配置文件end

vi /opt/mongodb/conf/shard3c.conf
#配置文件内容
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/shard3c/log/shard1.log
storage:
dbPath: /opt/mongodb/shard3c/data
journal:
    enabled: true
wiredTiger:
    engineConfig:
       cacheSizeGB: 20
processManagement:
fork: true
pidFilePath: /opt/mongodb/shard3c/log/shard1.pid
net:
port: 27033
bindIp: 0.0.0.0
replication:
    replSetName: shard3
sharding:
    clusterRole: shardsvr
#配置文件end

启动三台服务器的shard3 server

mongod -f /opt/mongodb/conf/shard3a.conf
mongod -f /opt/mongodb/conf/shard3b.conf
mongod -f /opt/mongodb/conf/shard3c.conf
登陆任意一台服务器，初始化副本集

mongo --port 27031
#使用admin数据库
use admin
#定义副本集配置
config ={_id : "shard3",members : [
{_id : 0, host : "127.0.0.1:27031" },
{_id : 1, host : "127.0.0.1:27032" },
{_id : 2, host : "127.0.0.1:27033"}
]
}
#初始化副本集配置
rs.initiate(config);
设置第三个分片副本集结束-------------------------------

4、配置路由服务器 mongos

先启动配置服务器和分片服务器,后启动路由实例:（三台机器）

vi /opt/mongodb/conf/mongos1.conf
##配置开始
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/mongos1/log/mongos.log
processManagement:
fork: true
net:
port: 20001
bindIp: 0.0.0.0
sharding:
configDB: configs/127.0.0.1:21001,127.0.0.1:21002,127.0.0.1:21003
##结束配置

vi /opt/mongodb/conf/mongos2.conf
##配置开始
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/mongos2/log/mongos.log
processManagement:
fork: true
net:
port: 20002
bindIp: 0.0.0.0
sharding:
configDB: configs/127.0.0.1:21001,127.0.0.1:21002,127.0.0.1:21003
##结束配置

vi /opt/mongodb/conf/mongos3.conf
##配置开始
systemLog:
destination: file
logAppend: true
path: /opt/mongodb/mongos3/log/mongos.log
processManagement:
fork: true
net:
port: 20003
bindIp: 0.0.0.0
sharding:
configDB: configs/127.0.0.1:21001,127.0.0.1:21002,127.0.0.1:21003
##结束配置

启动三台服务器的mongos server

mongos -f /opt/mongodb/conf/mongos1.conf
mongos -f /opt/mongodb/conf/mongos2.conf
mongos -f /opt/mongodb/conf/mongos3.conf

5、启用分片

目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到mongos路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。

登陆任意一台mongos

mongo --port 20001
#使用admin数据库
use admin
#串联路由服务器与分配副本集
sh.addShard("shard1/127.0.0.1:27011,127.0.0.1:27012,127.0.0.1:27013")
sh.addShard("shard2/127.0.0.1:27021,127.0.0.1:27022,127.0.0.1:27023")
sh.addShard("shard3/127.0.0.1:27031,127.0.0.1:27032,127.0.0.1:27033")
#查看集群状态
sh.status()

6、测试

目前配置服务、路由服务、分片服务、副本集服务都已经串联起来了，但我们的目的是希望插入数据，数据能够自动分片。连接在mongos上，准备让指定的数据库、指定的集合分片生效。

#指定testdb分片生效
db.runCommand( { enablesharding :"testdb"});
#指定数据库里需要分片的集合和片键
db.runCommand( { shardcollection : "testdb.table1",key : {id: 1} } )

我们设置testdb的 table1 表需要分片，根据 id 自动分片到 shard1 ，shard2，shard3 上面去。要这样设置是因为不是所有mongodb 的数据库和表都需要分片！

测试分片配置结果

mongo 127.0.0.1:20001
#使用testdb
use testdb;
#插入测试数据
for (var i = 1; i <= 1000; i++) {db.table1.save({id:i,"test1":"testval1"});}

#查看分片情况如下，部分无关信息省掉了
db.table1.stats();

{
        "sharded" : true,
        "ns" : "testdb.table1",
        "count" : 100000,
        "numExtents" : 13,
        "size" : 5600000,
        "storageSize" : 22372352,
        "totalIndexSize" : 6213760,
        "indexSizes" : {
                "_id_" : 3335808,
                "id_1" : 2877952
        },
        "avgObjSize" : 56,
        "nindexes" : 2,
        "nchunks" : 3,
        "shards" : {
                "shard1" : {
                        "ns" : "testdb.table1",
                        "count" : 42183,
                        "size" : 0,
                        ...
                        "ok" : 1
                },
                "shard2" : {
                        "ns" : "testdb.table1",
                        "count" : 38937,
                        "size" : 2180472,
                        ...
                        "ok" : 1
                },
                "shard3" : {
                        "ns" : "testdb.table1",
                        "count" :18880,
                        "size" : 3419528,
                        ...
                        "ok" : 1
                }
        },
        "ok" : 1
}

可以看到数据分到3个分片，各自分片数量为： shard1 “count” : 42183，shard2 “count” : 38937，shard3 “count” : 18880。已经成功了！

后期运维,启动关闭
mongodb的启动顺序是，先启动配置服务器，在启动分片，最后启动mongos.

mongod -f /opt/mongodb/conf/config1.conf
mongod -f /opt/mongodb/conf/config2.conf
mongod -f /opt/mongodb/conf/config3.conf
mongod -f /opt/mongodb/conf/shard1a.conf
mongod -f /opt/mongodb/conf/shard1b.conf
mongod -f /opt/mongodb/conf/shard1c.conf
mongod -f /opt/mongodb/conf/shard2a.conf
mongod -f /opt/mongodb/conf/shard2b.conf
mongod -f /opt/mongodb/conf/shard2c.conf
mongod -f /opt/mongodb/conf/shard3a.conf
mongod -f /opt/mongodb/conf/shard3b.conf
mongod -f /opt/mongodb/conf/shard3c.conf
mongos -f /opt/mongodb/conf/mongos1.conf
mongos -f /opt/mongodb/conf/mongos2.conf
mongos -f /opt/mongodb/conf/mongos3.conf

关闭时，直接killall杀掉所有进程
ps -aux |grep mongo
killall mongod
killall mongos

增加鉴权功能

1添加管理员帐号
mongo 127.0.0.1:20001
use admin
db.createUser(
        {
            user:"root2",
            pwd:"Qwe123!@#",
            roles:[{role:"root",db:"admin"}]
        }
    )

全部关闭mongodb

2依次在每台机器上的mongod（注意是所有的mongod不是mongos）的配置文件中加入下面一段配置。如我在10.12.40.83上的config server，shard1，shard2，shard3都加入下面的配置文件

security:
keyFile: /opt/mongodb/keyFile.file
authorization: enabled

3依次在每台机器上的mongos配置文件中加入下面一段配置。如我在10.12.40.83上的mongos配置文件中加入上面的一段配置

security:
keyFile: /opt/mongodb/keyFile.file

启动mongodb

测试鉴权效果
mongo 127.0.0.1:20001
use admin
show collections
db.auth("root", "Qwe123!@#")

经测试鉴权的配置必须要等mongodb初始化并添加管理员帐号后才可添加，不然会导致登录不上去
实际部署过程中，为了提高效率，复制了2个版本的配置文件
/opt/mongodb/conf/ 下的配置，没加鉴权
/opt/mongodb.auth/conf/下的配置，加了鉴权
第一次初始化时执行/opt/mongodb/conf/下的配置，添加管理员用户
以后启动mongodb时启动/opt/mongodb.auth/conf/下的配置

相关启动命令：
mongod -f /opt/mongodb/conf/config1.conf
mongod -f /opt/mongodb/conf/config2.conf
mongod -f /opt/mongodb/conf/config3.conf
mongod -f /opt/mongodb/conf/shard1a.conf
mongod -f /opt/mongodb/conf/shard1b.conf
mongod -f /opt/mongodb/conf/shard1c.conf
mongod -f /opt/mongodb/conf/shard2a.conf
mongod -f /opt/mongodb/conf/shard2b.conf
mongod -f /opt/mongodb/conf/shard2c.conf
mongod -f /opt/mongodb/conf/shard3a.conf
mongod -f /opt/mongodb/conf/shard3b.conf
mongod -f /opt/mongodb/conf/shard3c.conf
mongos -f /opt/mongodb/conf/mongos1.conf
mongos -f /opt/mongodb/conf/mongos2.conf
mongos -f /opt/mongodb/conf/mongos3.conf

mongod -f /opt/mongodb.auth/conf/config1.conf
mongod -f /opt/mongodb.auth/conf/config2.conf
mongod -f /opt/mongodb.auth/conf/config3.conf

mongod -f /opt/mongodb.auth/conf/shard1a.conf
mongod -f /opt/mongodb.auth/conf/shard2a.conf
mongod -f /opt/mongodb.auth/conf/shard3a.conf

mongod -f /opt/mongodb.auth/conf/shard1b.conf
mongod -f /opt/mongodb.auth/conf/shard1c.conf
mongod -f /opt/mongodb.auth/conf/shard2b.conf
mongod -f /opt/mongodb.auth/conf/shard2c.conf
mongod -f /opt/mongodb.auth/conf/shard3b.conf
mongod -f /opt/mongodb.auth/conf/shard3c.conf

mongos -f /opt/mongodb.auth/conf/mongos1.conf
mongos -f /opt/mongodb.auth/conf/mongos2.conf
mongos -f /opt/mongodb.auth/conf/mongos3.conf