MongoDB到MongoDB迁移-MongoShake

沉淅尘

已于 2024-10-12 10:14:51 修改

阅读量1.7k

点赞数 2

分类专栏： AWS MongoDB 文章标签： aws 运维 mongodb database

于 2022-03-04 10:20:03 首次发布

本文链接：https://blog.youkuaiyun.com/weixin_41335923/article/details/123270563

版权

AWS 同时被 2 个专栏收录

21 篇文章

订阅专栏

MongoDB

3 篇文章

订阅专栏

本文介绍了如何在Amazon Linux 2环境下，使用MongoShake进行MongoDB从副本集到副本集及分片集群的迁移。详细步骤包括MongoDB的安装、MongoShake的配置与迁移监控，以及全量和增量同步的检查。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

环境

操作系统：Amazon Linux 2

MongoDB版本：4.2.18

源端mongoDB：ip-10-10-1-12.ap-southeast-1.compute.internal或13.212.6.205

目标端MongoDB：ip-10-11-3-1.ap-southeast-1.compute.internal或13.212.248.207

MongoShake节点：ip-10-11-0-147.ap-southeast-1.compute.internal或13.212.91.106

一、副本集迁移

1.1 MongoDB安装（源端和目标端都需要安装）

MongoDB 源码下载地址：https://www.mongodb.com/download-center#community

wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-amazon2-4.2.18.tgz
tar -zxvf mongodb-linux-x86_64-amazon2-4.2.18.tgz
mv mongodb-linux-x86_64-amazon2-4.2.18.tgz /usr/local/mongodb4
export PATH=/usr/local/mongodb4/bin:$PATH

创建数据库目录

sudo mkdir -p /var/lib/mongo
sudo mkdir -p /var/log/mongodb
sudo chown `whoami` /var/lib/mongo 
sudo chown `whoami` /var/log/mongodb

以副本集方式启动MongoDB，–bind_ip开启远程访问

mongod --dbpath /var/lib/mongo --logpath /var/log/mongodb/mongod.log --fork --replSet rs0 --bind_ip 0.0.0.0
mongo localhost  # 进入mongo命令行
>rs.initiate()
>rs.conf()

mongod --dbpath /var/lib/mongo --logpath /var/log/mongodb/mongod.log --shutdown # 关闭mongodb

创建数据库和测试数据(在源端进入到mongo命令行中执行，目标端不需要执行)

use paas
db.createCollection("paas1")

for(i=1;i<=100000;i++) db.paas1.insert(
    {
        title: 'paas1_'+i,
        descriptionage: '(Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.)',
        by: 'paas1',
        url: 'http://www.t21_0121.com',
        tags: ['mongodb', 'database', 'NoSQL'],
        likes: i
    }
);

1.2 MongoShake安装和配置迁移

MongoShake版本：2.6.5

wget https://github.com/alibaba/MongoShake/releases/download/release-v2.6.5-20210723-1/mongo-shake-v2.6.5.tar.gz
tar -zxvf mongo-shake-v2.6.5.tar.gz
mv mongo-shake-v2.6.5.tar.gz mongoshake
cd mongoshake

修改collector.conf文件

mongo_urls = mongodb://ip-10-10-1-12.ap-southeast-1.compute.internal:27017 # 源端连接串信息，这里填公网IP或者私网DNS
sync_mode = all # all 表示全量+增量，full表示仅全量，incr表示仅增量
tunnel.address = mongodb://10.11.3.1:27017 # 目的端连接串信息，多个副本用逗号分隔不同的mongod
incr_sync.mongo_fetch_method = oplog # 如果希望以change stream拉取，该值需要配置change_stream，支持>=4.0.1版本。
incr_sync.tunnel.write_thread = 8 # 设置为"incr_sync.worker"的值一样8

启动

./collector.linux -conf=collector.conf &

查看MongoShake增量迁移监控(只有涉及到增量同步才可以查看)

./mongoshake-stat --port=9100

在这里插入图片描述

全量同步通过查看logs/collector.log文件日志，或者进入到目标端查看对应的数据库和数据

二、分片集群迁移

在这里插入图片描述

Shard:

用于存储实际的数据块，实际生产环境中一个shard server角色可由几台机器组个一个replica set承担，防止主机单点故障
Config Server:

mongod实例，存储了整个 ClusterMetadata，其中包括 chunk信息。
Query Routers:

前端路由，客户端由此接入，且让整个集群看上去像单一数据库，前端应用可以透明使用。

2.1 分片集群部署安装

下载与副本集一致，但是启动方式不同。这里一台EC2，通过不同目录来作为分片节点，搭建分片集群。

# 创建数据存储
mkdir -p /www/mongoDB/shard/s0
mkdir -p /www/mongoDB/shard/s1
mkdir -p /www/mongoDB/shard/s2
mkdir -p /www/mongoDB/shard/s3
# 创建日志存储
mkdir -p /www/mongoDB/shard/log
# 创建Sharding副本集 rs0
mongod --port 27020 --dbpath=/www/mongoDB/shard/s0 --logpath=/www/mongoDB/shard/log/s0.log -logappend --fork --shardsvr --replSet=rs0 --bind_ip_all
mongod --port 27021 --dbpath=/www/mongoDB/shard/s1 --logpath=/www/mongoDB/shard/log/s1.log -logappend --fork --shardsvr --replSet=rs0 --bind_ip_all
mongod --port 27022 --dbpath=/www/mongoDB/shard/s2 --logpath=/www/mongoDB/shard/log/s2.log -logappend --fork --shardsvr --replSet=rs1 --bind_ip_all
mongod --port 27023 --dbpath=/www/mongoDB/shard/s3 --logpath=/www/mongoDB/shard/log/s3.log -logappend --fork --shardsvr --replSet=rs1 --bind_ip_all
# 创建config目录，作为Metadata存储
mkdir -p /www/mongoDB/shard/config1
mkdir -p /www/mongoDB/shard/config2
# 创建Config副本集 conf
mongod --port 27100 -dbpath=/www/mongoDB/shard/config1 --logpath=/www/mongoDB/shard/log/config1.log --logappend --fork --configsvr --replSet=conf --bind_ip_all
mongod --port 27101 -dbpath=/www/mongoDB/shard/config2 --logpath=/www/mongoDB/shard/log/config2.log --logappend --fork --configsvr --replSet=conf --bind_ip_all

副本集初始化

mongo localhost:27020
>use admin
>rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'localhost:27020'}, {_id: 1, host: 'localhost:27021'}]})
>rs.isMaster()

mongo localhost:27022
>use admin
>rs.initiate({_id: 'rs1', members: [{_id: 0, host: 'localhost:27022'}, {_id: 1, host: 'localhost:27023'}]})
>rs.isMaster()

mongo localhost:27100
>use admin
>rs.initiate({_id: 'conf', members: [{_id: 0, host: 'localhost:27100'}, {_id: 1, host: 'localhost:27101'}]})
>rs.isMaster()

创建Route和设置分片

mongos --port 40000 --configdb conf/localhost:27100 --fork --logpath=/www/mongoDB/shard/log/route.log --logappend --bind_ip_all
mongo localhost:40000
> use admin
> db.runCommand({ addshard: 'rs0/localhost:27020,localhost:27021'})
> db.runCommand({ addshard: 'rs1/localhost:27022,localhost:27023'})
# 下面命令只在源端执行
> db.runCommand({ enablesharding: 'test'})
> db.runCommand({ shardcollection: 'test.user', key: {name: 1}})

创建测试数据（只在源端的mongo 命令行中执行）

use test
db.createCollection("paas")

for(i=1;i<=100000;i++) db.paas.insert(
    {
        title: 'paas_'+i,
        descriptionage: '(Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.)',
        by: 'paas',
        url: 'http://www.t21_0121.com',
        tags: ['mongodb', 'database', 'NoSQL'],
        likes: i
    }
);
use paas
db.createCollection("paas1")

for(i=1;i<=100000;i++) db.paas1.insert(
    {
        title: 'paas1_'+i,
        descriptionage: '(Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage. Amazon EC2 enables you to scale up or down to handle changes in requirements or spikes in popularity, reducing your need to forecast traffic.)',
        by: 'paas1',
        url: 'http://www.t21_0121.com',
        tags: ['mongodb', 'database', 'NoSQL'],
        likes: i
    }
);

2.2 MongoShake配置

前面已经安装过mongoShake，这里只需要修改collector.cof文件

mongo_urls = mongodb://13.212.6.205:27020,13.212.6.205:27021;mongodb://13.212.6.205:27022,13.212.6.205:27023 # 源端分片集
mongo_cs_url = mongodb://13.212.6.205:27100 # 源端config Master
mongo_s_url = mongodb://13.212.6.205:40000 # 源端Mongos地址
sync_mode = all # all 表示全量+增量，full表示仅全量，incr表示仅增量
tunnel.address = mongodb://10.11.3.1:27020,10.11.3.1:27021;mongodb://10.11.3.1:27022,10.11.3.1:27023 # 目的端分片集
incr_sync.mongo_fetch_method = oplog # 如果希望以change stream拉取，该值需要配置change_stream，支持>=4.0.1版本。
incr_sync.tunnel.write_thread = 8 # 设置为"incr_sync.worker"的值一样8

启动

./collector.linux -conf=collector.conf &

查看MongoShake增量迁移监控(只有涉及到增量同步才可以查看)

./mongoshake-stat --port=9100

在这里插入图片描述

全量同步通过查看logs/collector.log文件日志，或者进入到目标端查看对应的数据库和数据