异地容灾Drbd
实验目的:
掌握Drbd的理论与配置
实验理论:
Drbd功能:
把多台机器的硬盘组合成一个网络Raid,保证底层数据的同步,用在异地容灾
目前Drbd8版本只支持两个节点来做网络Raid,有个Drbd9版本,可以支持3或多个节点来做网络Raid(一台主、两台从)
Drbd模式:
主从关系:使用任意文件系统来格式化硬盘,主节点有数据会自动向辅节点同步数据,辅节点有数据不会向主节点同步数据
主主关系:只能使用Gfs文件系统来格式化硬盘,两边都是主主关系,主节点1有数据会自动向主节点2同步数据,主节点2有数据会自动向主节点1同步数据
Drbd协议:
A协议:数据节点1向节点2同步数据时,节点1只管把数据发出去就行,不考虑链路的状况,不考虑节点2是否接收得到数据
B协议:数据节点1向节点2同步数据时,节点1要考虑把数据发出去,要考虑链路的状况(确保把数据发到节点2上),不考虑节点2是否接收得到数据
C协议:数据节点1向节点2同步数据时,节点1要考虑把数据发出去,要考虑链路的状况,还要考虑节点2是否接收得到数据
系统默认策略是C策略
注意:
一、防火墙与Selinux都要关掉关掉,
二、红帽系统内核2.6.33版本之上的内核,会内置Drbd软件,在2.5.33内核版本之下的,需要你手动安装Drbd软件。红帽5内核是2.6.18,红帽6内核是2.6.32,我是手动安装Drbd软件的。
三、Drbd软件分为rpm包和源码包
实验步骤:
Drbd1:
Drbd1的IP地址是192.168.1.250
[root@drbd1 drbd.d]# vim /etc/hosts
192.168.1.250 drbd1.example.com
192.168.1.200 drbd2.example.com
[root@drbd1 ~]# fdisk /dev/sda
Command (m for help): n
First cylinder (6649-13054, default 6649):
Using default value 6649
Last cylinder or +size or +sizeM or +sizeK(6649-13054, default 13054): +1G
Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): 8e
Changed system type of partition 5 to 8e(Linux LVM)
Command (m for help): w
[root@drbd1 ~]# partprobe /dev/sda
[root@drbd1 ~]# pvcreate /dev/sda5
Writing physical volume data to disk "/dev/sda5"
Physical volume "/dev/sda5" successfully created
[root@drbd1 ~]# vgcreate vg01 /dev/sda5
Volume group "vg01" successfully created
[root@drbd1 ~]# lvcreate -l 100%free -n lv01 vg01
Logical volume "lv01" created
[root@drbd1 ~]# cd /home
[root@drbd1 home]# ls 这两个软件包都是64位的,你的红帽5系统也要是64位的,否则软件包装不上去,kmod软件包是安装Drbd模块的作用
drbd83-8.3.15-2.el5.centos.x86_64.rpm kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm
[root@drbd2 home]# rpm -ivh drbd83-8.3.15-2.el5.centos.x86_64.rpm
warning:drbd83-8.3.15-2.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, key IDe8562897
Preparing... ########################################### [100%]
1:drbd83 ########################################### [100%]
[root@drbd2 home]# rpm -ivh kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm
warning:kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, keyID e8562897
Preparing... ###########################################[100%]
1:kmod-drbd83 ########################################### [100%]
[root@drbd1 home]# modprobe drbd
modprobe drbd 当前运行drbd模块,可以启动Drbd的服务
[root@drbd1 home]#echo “modprobe drbd” >> /etc/rc.local
把这个命令放到开机文件中,保证drbd模块重启系统后也在运行
[root@drbd1 home]# vim /etc/drbd.conf
#
# please have a a look at the exampleconfiguration file in
# /usr/share/doc/drbd83/drbd.conf
#
[root@drbd1 home]# cp /usr/share/doc/drbd83-8.3.15/drbd.conf /etc/drbd.conf
cp: overwrite `/etc/drbd.conf'? yes
复制/usr目录下的drbd.conf文件,是让/etc/drbd.conf文件生效
[root@drbd1 home]# vim /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include"drbd.d/global_common.conf";
include"drbd.d/*.res"; 在/etc/drbd.d目录下要有这两种文件global_common.conf和*.res
[root@drbd1 ~]# cd /etc/drbd.d/
[root@drbd1 drbd.d]# ls
global_common.conf
[root@drbd1 ~]# vim /etc/drbd.d/global_common.conf
global { global_common.conf文件里只需要下面信息就可以了
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
使用那种协议?A?还是B?还是C?默认协议是C
common {
protocol C;
}
[root@drbd1 drbd.d]# vim a.res
resource a0 {
disk /dev/vg01/lv01; 分号,注意分号!
device /dev/drbd0;
meta-disk internal; 数据进行网络连接
on drbd1.example.com {
address 192.168.1.250:7788;
}
on drbd2.example.com {
address 192.168.1.200:7788;
}
}
[root@drbd1 drbd.d]# drbdadm create-md a0 创建drbd数据块
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfullycreated.
[root@drbd1 drbd.d]# drbdadm adjust a0
Drbd2:
Drbd2的IP地址为192.168.1.200
[root@drbd2 ~]# vim /etc/hosts
192.168.1.250 drbd1.example.com
192.168.1.200 drbd2.example.com
[root@drbd2 ~]# fdisk /dev/sda
Command (m for help): n
First cylinder (6649-13054, default 6649):
Using default value 6649
Last cylinder or +size or +sizeM or +sizeK(6649-13054, default 13054): +1G
Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): 8e
Changed system type of partition 5 to 8e(Linux LVM)
Command (m for help): w
[root@drbd2 ~]# partprobe /dev/sda
[root@drbd2 ~]# pvcreate /dev/sda5
Writing physical volume data to disk "/dev/sda5"
Physical volume "/dev/sda5" successfully created
[root@drbd2 ~]# vgcreate vg01 /dev/sda5
Volume group "vg01" successfully created
[root@drbd2 ~]# lvcreate -l 100%free -n lv01 vg01
Logical volume "lv01" created
[root@drbd2 ~]# cd /home
[root@drbd2 home]# ls
drbd83-8.3.15-2.el5.centos.x86_64.rpm kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm
[root@drbd2 home]# rpm -ivh drbd83-8.3.15-2.el5.centos.x86_64.rpm
warning:drbd83-8.3.15-2.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, key IDe8562897
Preparing... ########################################### [100%]
packagedrbd83-8.3.15-2.el5.centos.x86_64 is already installed
[root@drbd2 home]# rpm -ivh kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm
warning:kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, keyID e8562897
Preparing... ########################################### [100%]
packagekmod-drbd83-8.3.15-3.el5.centos.x86_64 is already installed
[root@drbd2 home]# modprobe drbd
[root@drbd2 home]#echo “modprobe drbd” >> /etc/rc.local
[root@drbd2 home]# cp /usr/share/doc/drbd83-8.3.15/drbd.conf /etc/drbd.conf
cp: overwrite `/etc/drbd.conf'? yes
[root@drbd2 home]# cd /etc/drbd.d/
[root@drbd2 drbd.d]# ls
global_common.conf
[root@drbd1 ~]# vim /etc/drbd.d/global_common.conf
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
}
[root@drbd2 drbd.d]# vim a.res
resource a0 {
disk /dev/vg01/lv01;
device /dev/drbd0;
meta-disk internal;
on drbd1.example.com {
address 192.168.1.250:7788;
}
on drbd2.example.com {
address 192.168.1.200:7788;
}
}
[root@drbd2 drbd.d]# drbdadm create-md a0 --force
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfullycreated.
[root@drbd2 drbd.d]# drbdadm adjust a0
Drbd1:
[root@drbd1 drbd.d]# service drbd status drbd起来之后两个节点默认都是从关系
drbd driver loaded OK; device status: 需要你设置某个节点为主
version: 8.3.15 (api:88/proto:86-97)
GIT-hash:0ce4d235fc02b5c53c1c52c53433d11a694eab8c build bymockbuild@builder10.centos.org, 2013-03-27 16:01:26
m:res cs ro ds p mounted fstype
0:a0 Connected Secondary/Secondary Inconsistent/Inconsistent C
[root@drbd1 drbd.d]# drbdsetup 0 primary --force 在Drbd1节点上敲drbdsetup 0primary --force就是设置Drbd1为主节点,如果drbdsetup 0 primary --force命令不生效敲drbdadm primary a0命令也可以设置Drbd1节点为主节点
[root@drbd1 drbd.d]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.15 (api:88/proto:86-97)
GIT-hash:0ce4d235fc02b5c53c1c52c53433d11a694eab8c build bymockbuild@builder10.centos.org, 2013-03-27 16:01:26
m:res cs ro ds p mounted fstype
... sync'ed: 44.0% (59224/102360)K
0:a0 SyncSource Primary/Secondary UpToDate/Inconsistent C
[root@drbd1 drbd.d]#watch -n .5 cat /proc/drbd 查看Drbd1、2两节点的同步状态
[root@drbd1 drbd.d]#mkfs.ext3 /dev/drbd0 Drbd1为主,就在是主的节点对/dev/drbd0设备格式化和挂载,辅的节点不要格式化、挂载
[root@drbd1 drbd.d]#mount /dev/drbd0 /mnt
Drbd 3个节点:
问题:前面我们说道Drbd只能对两个节点做数据同步,那怎么做到3节点的数据同步了?
把Drbd1和Drbd2配置Drbd软件进行数据同步,得到设备/dev/drbd0,这时我们可以把Drbd1和Drbd2看成是一台机器,这台机器拥有设备/dev/drbd0,这台机器要有一个VIP地址,在Drbd1和Drbd2都要有一个网卡,两个网卡的IP一样的,这个IP就是这台机器的Vip,这台机器和Drbd3在做Drbd配置,进行数据同步,得到设备/dev/drbd10,使用这种堆栈的方式,甚至可以做到4、5节点数据同步,但这并没有违背Drbd只能对两节点进行数据同步。
Drbd1、2、3同样的配置:
[root@Drbd3 student]# vim /etc/hosts
192.168.1.250 drbd1.example.com
192.168.1.200 drbd2.example.com
192.168.1.150 drbd3.example.com
[root@drbd1 ~]# fdisk /dev/sda
Command (m for help): n
First cylinder (6649-13054, default 6649):
Using default value 6649
Last cylinder or +size or +sizeM or +sizeK(6649-13054, default 13054): +1G
Command (m for help): t
Partition number (1-5): 5
Hex code (type L to list codes): 8e
Changed system type of partition 5 to 8e(Linux LVM)
Command (m for help): w
[root@drbd1 ~]# partprobe /dev/sda
[root@drbd1 ~]# pvcreate /dev/sda5
Writing physical volume data to disk "/dev/sda5"
Physical volume "/dev/sda5" successfully created
[root@drbd1 ~]# vgcreate vg01 /dev/sda5
Volume group "vg01" successfully created
[root@drbd1 ~]# lvcreate -l 100%free -n lv01 vg01
Logical volume "lv01" created
[root@drbd1 home]# rpm -ivh drbd83-8.3.15-2.el5.centos.x86_64.rpm
warning:drbd83-8.3.15-2.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, key IDe8562897
Preparing... ########################################### [100%]
1:drbd83 ###########################################[100%]
[root@drbd1 home]# rpm -ivh kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm
warning:kmod-drbd83-8.3.15-3.el5.centos.x86_64.rpm: Header V3 DSA signature: NOKEY, keyID e8562897
Preparing... ###########################################[100%]
1:kmod-drbd83 ########################################### [100%]
[root@drbd1 home]# modprobe drbd
[root@Drbd3 student]# cp /usr/share/doc/drbd83-8.3.15/drbd.conf /etc/drbd.conf
cp: overwrite `/etc/drbd.conf'? yes
[root@Drbd3 student]# vim /etc/drbd.d/global_common.conf
global {
usage-countyes;
#minor-count dialog-refresh disable-ip-verification
}
common {
protocolC;
}
[root@Drbd3 student]# vim /etc/drbd.d/a.res
resource a0 {
device /dev/drbd0;
disk /dev/vg01/lv01;
meta-disk internal;
on drbd1.example.com {
address 192.168.1.250:7788;
}
on drbd2.example.com {
address 192.168.1.200:7788;
}
}
resource a1 {
on drbd3.example.com {
device /dev/drbd10;
disk /dev/vg01/lv01;
meta-disk internal;
address 192.168.1.150:7788;
}
stacked-on-top-of a0 { 堆栈的方式,Drbd1和Drbd2组合的一台机器
device /dev/drbd10;
address 192.168.1.100:7788; 填写组合机器的Vip
}
}
Drbd1:
[root@drbd1drbd.d]# ifconfigeth0:0 192.168.1.100 netmask 255.255.255.0 up VIP地址
[root@drbd1drbd.d]# drbdadm create-md a0
Writingmeta data...
initializingactivity log
NOTinitialized bitmap
Newdrbd meta data block successfully created.
[root@drbd1drbd.d]# drbdadm adjust a0
Drbd2:
[root@drbd2drbd.d]# ifconfig eth0:0 192.168.1.100 netmask 255.255.255.0 up VIP地址
[root@drbd2drbd.d]# drbdadm create-md a0
Writingmeta data...
initializingactivity log
NOTinitialized bitmap
Newdrbd meta data block successfully created.
[root@drbd2drbd.d]# drbdadm adjust a0
Drbd1和Drbd2做完了Drbd配置,设置了Vip,可以看成是一台机器了,我把Drbd1设置为主
[root@drbd1drbd.d]# drbdsetup 0 primary --force
堆栈的主机和Drbd3主Drbd配置:
[root@Drbd3student]# drbdadm create-md a1
Writingmeta data...
initializingactivity log
NOTinitialized bitmap
Newdrbd meta data block successfully created.
[root@Drbd3student]# drbdadm adjust a1
Drbd1: Drbd1和Drbd2那个是主,是主的主机就代表被堆栈的主机
[root@drbd1drbd.d]# drbdadm create-md a1 -S 一定要加-S,-S表示堆栈
Writingmeta data...
initializingactivity log
NOTinitialized bitmap
Newdrbd meta data block successfully created.
[root@drbd1drbd.d]# drbdadm adjust a1 -S
[root@drbd1drbd.d]# drbdsetup 10 primary --force 设置堆栈的主机为主,Drbd3为辅
[root@drbd1drbd.d]# service drbd status
drbddriver loaded OK; device status:
version:8.3.15 (api:88/proto:86-97)
GIT-hash:0ce4d235fc02b5c53c1c52c53433d11a694eab8c build bymockbuild@builder10.centos.org, 2013-03-27 16:01:26
m:res cs ro ds p mounted fstype
0:a0 Connected Primary/Secondary UpToDate/UpToDate C
... sync'ed: 12.0% (91440/102320)K
10:a1^^0 SyncSource Primary/Secondary UpToDate/Inconsistent C
记得格式化和挂载!