昨天有个客户那边重装了一台机器,但是没有拔掉光纤线,然后做选择的时候还把那个LUN给初始化了,导致新的服务器加到资源池里存储一直挂不上。有几个虚拟机一迁移就失败了,还开不了机。SSH上去看,磁盘已经被格式化了,还创建了一个分区。
以1.36和1.58为列 说一下解决步骤
1.36上是主节点,上面有几个虚拟机在运行。
在1.36上执行下面一系列命令
首先是fdisk -l 因为有两条路径 所以能看到/dev/sdb和/dev/sdc是一样的,与正常的SR的磁盘的区别是多了一个GPT分区
[root@localhost ~]# fdisk -l Disk /dev/sda: 2000.3 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 522 4192933+ 83 Linux /dev/sda2 523 783 2096482+ 83 Linux /dev/sda3 784 243201 1947222585 8e Linux LVM WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted. WARNING: The size of this disk is 2.6 TB (2576980377600 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Disk /dev/sdb: 2576.9 GB, 2576980377600 bytes 255 heads, 63 sectors/track, 313300 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 267350 2147483647+ ee EFI GPT WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted. WARNING: The size of this disk is 2.6 TB (2576980377600 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Disk /dev/sdc: 2576.9 GB, 2576980377600 bytes 255 heads, 63 sectors/track, 313300 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 267350 2147483647+ ee EFI GPT |
运行pvscan 可以看到Physical Volume已经不见了
[root@localhost ~]# pvscan PV /dev/sda3 VG VG_XenStorage-04c0df5f-b878-9d62-5157-649b0ebc8980 lvm2 [1.81 TB / 1.81 TB free] Total: 1 [1.81 TB] / in use: 1 [1.81 TB] / in no VG: 0 [0 ] |
9ba32798-7dc6-0b2b-8abc-645192a9203a这个ID是我们出问题的存储(SR)的UUID
[root@localhost ~]# xe sr-list uuid ( RO) : ee931812-2d09-fd43-8b42-73fd4f512512 name-label ( RW): DVD drives name-description ( RW): Physical DVD drives host ( RO): XenServer58 type ( RO): udev content-type ( RO): iso uuid ( RO) : 04c0df5f-b878-9d62-5157-649b0ebc8980 name-label ( RW): Local storage name-description ( RW): host ( RO): localhost.localdomain type ( RO): lvm content-type ( RO): user uuid ( RO) : 9ba32798-7dc6-0b2b-8abc-645192a9203a name-label ( RW): 硬件 HBA èæç£çåå¨ name-description ( RW): 硬件 HBA SR [DELL - 2CC001B] host ( RO): <shared> type ( RO): lvmohba content-type ( RO): uuid ( RO) : 396812b6-e90b-0761-50a3-b08e57c19453 name-label ( RW): Local storage name-description ( RW): host ( RO): XenServer58 type ( RO): lvm content-type ( RO): user .... |
在/etc/lvm/backup中系统自动备份的lvm信息
[root@localhost ~]# cd /etc/lvm/backup/ [root@localhost backup]# ls -l total 8 -rw------- 1 root root 1311 Jan 17 14:02 VG_XenStorage-04c0df5f-b878-9d62-5157-649b0ebc8980 -rw------- 1 root root 3454 Jan 17 14:02 VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a |
因为这个LUN还在被1.36上面的虚拟机使用,pv还在被占用,所以我们把文件拷贝到1.58上,去1.58上执行之后的操作
[root@localhost backup]# scp -r /etc/lvm/backup/ root@192.168.1.58:/root/lvm_backcup The authenticity of host '192.168.1.58 (192.168.1.58)' can't be established. RSA key fingerprint is 21:66:46:86:60:79:12:9a:23:f7:99:34:97:12:e7:a8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.1.58' (RSA) to the list of known hosts. root@192.168.1.58's password: VG_XenStorage-04c0df5f-b878-9d62-5157-649b0ebc8980 100% 1311 1.3KB/s 00:00 VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a 100% 3454 3.4KB/s 00:00 |
在1.58上执行下面的操作
首先删除多出来的/dev/sdb1分区
[root@localhost ~]# fdisk /dev/sdb
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted. The number of cylinders for this disk is set to 313300. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) WARNING: The size of this disk is 2.6 TB (2576980377600 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Command (m for help): d Selected partition 1 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. [root@localhost ~]# fdisk -l Disk /dev/sda: 146.8 GB, 146815733760 bytes 255 heads, 63 sectors/track, 17849 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 522 4192933+ 83 Linux /dev/sda2 523 783 2096482+ 83 Linux /dev/sda3 784 17849 137082645 8e Linux LVM WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted. WARNING: The size of this disk is 2.6 TB (2576980377600 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Disk /dev/sdb: 2576.9 GB, 2576980377600 bytes 255 heads, 63 sectors/track, 313300 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted. WARNING: The size of this disk is 2.6 TB (2576980377600 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT). Disk /dev/sdc: 2576.9 GB, 2576980377600 bytes 255 heads, 63 sectors/track, 313300 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System [root@localhost ~]# |
查看lvm备份文件找到physical volume的id,这里是"wQbSMU-CAVF-8P1K-aruD-h8Ts-btVf-jdu0nM"
[root@localhost ~]# cd /root/lvm_backcup/ [root@localhost lvm_backcup]# ls -l total 8 -rw------- 1 root root 1311 Jan 16 16:13 VG_XenStorage-04c0df5f-b878-9d62-5157-649b0ebc8980 -rw------- 1 root root 3454 Jan 16 16:13 VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a [root@localhost lvm_backcup]# cat VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a | grep physical_volumes -A3 physical_volumes { pv0 { id = "wQbSMU-CAVF-8P1K-aruD-h8Ts-btVf-jdu0nM" |
重建physical volume
注意因为没有启用多路径,所以使用/dev/sdb或者/dev/sdc都可以。如果之前启用了多路径需要使用目录
/dev/mapper/<scsi id> or /dev/disk/mpInuse/<scsi id>.
[root@localhost lvm_backcup]# pvcreate --uuid "wQbSMU-CAVF-8P1K-aruD-h8Ts-btVf-jdu0nM" --restorefile /root/lvm_backcup/VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a /dev/sdb Couldn't find device with uuid wQbSMU-CAVF-8P1K-aruD-h8Ts-btVf-jdu0nM. Physical volume "/dev/sdb" successfully created [root@localhost lvm_backcup]# pvscan Found duplicate PV wQbSMUCAVF8P1KaruDh8TsbtVfjdu0nM: using /dev/sdc not /dev/sdb PV /dev/sda3 VG VG_XenStorage-396812b6-e90b-0761-50a3-b08e57c19453 lvm2 [130.72 GB / 130.71 GB free] PV /dev/sdc lvm2 [2.34 TB] Total: 2 [2.47 TB] / in use: 1 [130.72 GB] / in no VG: 1 [2.34 TB] |
重建Volume Group
先进行测试
[root@localhost lvm_backcup]# vgcfgrestore VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a --test -f /root/lvm_backcup/VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a Test mode: Metadata will NOT be updated and volumes will not be (de)activated. Found duplicate PV wQbSMUCAVF8P1KaruDh8TsbtVfjdu0nM: using /dev/sdc not /dev/sdb Restored volume group VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a |
执行重建命令
[root@localhost lvm_backcup]# vgcfgrestore VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a -f /root/lvm_backcup/VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a
Found duplicate PV wQbSMUCAVF8P1KaruDh8TsbtVfjdu0nM: using /dev/sdc not /dev/sdb
Restored volume group VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a |
检查效果
[root@localhost lvm_backcup]# pvscan
Found duplicate PV wQbSMUCAVF8P1KaruDh8TsbtVfjdu0nM: using /dev/sdc not /dev/sdb
PV /dev/sdc VG VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a lvm2 [2.34 TB / 2.26 TB free] PV /dev/sda3 VG VG_XenStorage-396812b6-e90b-0761-50a3-b08e57c19453 lvm2 [130.72 GB / 130.71 GB free] Total: 2 [2.47 TB] / in use: 2 [2.47 TB] / in no VG: 0 [0 ] [root@localhost lvm_backcup]# vgscan Reading all physical volumes. This may take a while... Found duplicate PV wQbSMUCAVF8P1KaruDh8TsbtVfjdu0nM: using /dev/sdc not /dev/sdb Found volume group "VG_XenStorage-9ba32798-7dc6-0b2b-8abc-645192a9203a" using metadata type lvm2 Found volume group "VG_XenStorage-396812b6-e90b-0761-50a3-b08e57c19453" using metadata type lvm2 |
在Console上修复存储,可以成功挂载了。关闭虚拟机,重启所有的服务器,之后再启动虚拟机就没问题了。