-
Server rebooted during lvm snapshot and put volume group in inconsistent state.
# vgs VG #PV #LV #SN Attr VSize VFree vg0 1 3 0 wz--n- 229.66G 9.66G vg1 2 3 0 wz-pn- 279.46G 9.46G
Note the "p" attribute in vg1 which put the group in partial mode.
To resolve, I first vgreduce then re-created the volume group from a good known archive:# vgreduce --removemissing --force vg1 vsrv31_var_lib_mysql is expected to have only one segment using it, while it has 0 Failed to find mirror_seg for vsrv31_var_lib_mysql vsrv31_var_log_mysql is expected to have only one segment using it, while it has 0 Failed to find mirror_seg for vsrv31_var_log_mysql vsrv01_var_log is expected to have only one segment using it, while it has 0 Failed to find mirror_seg for vsrv01_var_log Wrote out consistent volume group vg1Upon listing the volume groups, it automatically removed /dev/sdd1 from vg1.
# vgs WARNING: Inconsistent metadata found for VG vg1 - updating to use version 1675 Removing PV /dev/sdd1 (2tfAMr-erXr-fm1G-dicV-W08Q-eqExtended vg1 to included sdd1:g6-CL8nbH) that no longer belongs to VG vg1 VG #PV #LV #SN Attr VSize VFree vg0 1 3 0 wz--n- 229.66G 9.66G vg1 1 0 0 wz--n- 139.73G 139.73G #pvs PV VG Fmt Attr PSize PFree /dev/md2 vg0 lvm2 a- 229.66G 9.66G /dev/md3 vg1 lvm2 a- 139.73G 139.73G /dev/sdd1 lvm2 -- 139.73G 139.73G
# vgextend vg1 /dev/sdd1 Volume group "vg1" successfully extended # pvs PV VG Fmt Attr PSize PFree /dev/md2 vg0 lvm2 a- 229.66G 9.66G /dev/md3 vg1 lvm2 a- 139.73G 139.73G /dev/sdd1 vg1 lvm2 a- 139.73G 139.73G # vgs VG #PV #LV #SN Attr VSize VFree vg0 1 3 0 wz--n- 229.66G 9.66G vg1 2 0 0 wz--n- 279.46G 279.46G # lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert lv0_145186_var_www vg0 -wi-ao 16.00G lv0_swap vg0 -wi-ao 4.00G lv0_vz vg0 -wi-ao 200.00GThen did a listing of vg1 configs from archives (files located in /etc/lvm/archive):
# vgcfgrestore -l vg1 File: /etc/lvm/archive/vg1_02156.vgRestored the last known good archive:VG name: vg1 Description: Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql' Backup Time: Thu Feb 27 02:39:26 2014 File: /etc/lvm/archive/vg1_02157.vg VG name: vg1 Description: Created *before* executing '/usr/sbin/lvremove -f /dev/vg1/snap_vsrv31' Backup Time: Thu Feb 27 07:35:34 2014 File: /etc/lvm/archive/vg1_02158.vg VG name: vg1 Description: Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql' Backup Time: Fri Feb 28 02:37:38 2014
# vgcfgrestore -f /etc/lvm/archive/vg1_02158.vg vg1 Restored volume group vg1 # vgs VG #PV #LV #SN Attr VSize VFree vg0 1 3 0 wz--n- 229.66G 9.66G vg1 2 3 0 wz--n- 279.46G 9.46G # pvs PV VG Fmt Attr PSize PFree /dev/md2 vg0 lvm2 a- 229.66G 9.66G /dev/md3 vg1 lvm2 a- 139.73G 0 /dev/sdd1 vg1 lvm2 a- 139.73G 9.46G # lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert lv0_145186_var_www vg0 -wi-ao 16.00G lv0_swap vg0 -wi-ao 4.00G lv0_vz vg0 -wi-ao 200.00G vsrv01_var_log vg1 -wi--- 10.00G vsrv31_var_lib_mysql vg1 -wi--- 250.00G vsrv31_var_log_mysql vg1 -wi--- 10.00GActivated the volume group:
# vgchange -ay vg1 3 logical volume(s) in volume group "vg1" now active # lvs LV VG Attr LSize Origin Snap% Move Log Copy% Convert lv0_145186_var_www vg0 -wi-ao 16.00G lv0_swap vg0 -wi-ao 4.00G lv0_vz vg0 -wi-ao 200.00G vsrv01_var_log vg1 -wi-a- 10.00G vsrv31_var_lib_mysql vg1 -wi-a- 250.00G vsrv31_var_log_mysql vg1 -wi-a- 10.00GI then got around to force check the logical volumes prior to mounting it and luckily all data was intact.
Bookmark/Search this post with
- sandip's blog
- Login or register to post comments