This procedure has been tested in the lab. Please test it yourself before using it in a customer's environment and also make sure you have a copy of the current streams before you replace or change anything
The Applogic controller meta volume is a crucial element in a grid operation. Basically, the meta volume contains the following structure:
drwxrwxr-x 95 500 500 4096 Mar 19 16:54 apps drwxrwxr-x 10 500 500 4096 May 25 2012 catalogs lrwxrwxrwx 1 root root 27 Apr 2 2012 cluster.desc -> /etc/applogic/applogic.conf drwxr-xr-x 3 root root 4096 Mar 14 09:23 conf drwxr-xr-x 2 root root 4096 Apr 2 2012 cores drwxr-xr-x 2 root root 4096 Apr 2 2012 ha drwxrwxr-x 2 500 500 12288 Mar 19 09:34 license drwxr-xr-x 2 root root 4096 Apr 2 2012 logs drwx------ 2 root root 16384 Apr 2 2012 lost+found drwxr-xr-x 2 root root 4096 Apr 2 2012 picture drwxr-xr-x 4 root root 4096 Mar 6 17:43 rbac drwxr-xr-x 2 root root 4096 Jun 2 2010 scripts drwxr-xr-x 4 500 500 4096 Mar 19 17:28 state drwxr-xr-x 2 root root 4096 Mar 19 13:24 stats drwx------ 2 500 500 4096 Mar 19 12:18 users drwxrwxr-x 2 root root 36864 Mar 19 16:54 volumes
It is important to consider the following points:
The apps directory within it contains an entry for each one of the applications running at a certain point of time in the controller, which is used to restart applications on controller restart.
The srvs dir contains an entry for each server in the grid with its specific state at a given point in time, so that controller may pick it up upon restarting and check the applications running there. Incorrect configuration means 3tgridctl will fail on controller startup and you will not be able to see in which state applications are.
Finally there is another directory called vols containing the status of the mounted volumes at a given point in time.
The state file indicates the status of high availability at a certain point in time
With this in mind it is clear that a damaged or lost meta volume for a controller may render totally unusable a grid and therefore it is paramount to have a procedure to recover it or to repair it in case of need.
Contrarily to the boot volume, up to the present version (3.7.7), the meta volume of AppLogic is not partitioned, so it is in fact slightly easier to manage than the controller boot volume.
Applogic takes regular backup of the meta volume. These backups are stored as grid metadata backups in the BFC under directory /opt/bfc/grid_metadata_backup
The present document outlines the steps necessary to attempt repair of a controller meta volume.
It must be noted that a corrupted or missing controller meta volume will not prevent the controller from starting. The controller meta volume is mounted as:
[root@loda31tg2 ~]# df -kh Filesystem Size Used Avail Use% Mounted on /dev/hda1 1.2G 816M 329M 72% / /dev/hdb 1008M 49M 908M 6% /var/applogic none 385M 0 385M 0% /dev/shm 127.0.0.1:/var/local/lib/3trsh 1.2G 816M 329M 72% /vol 127.0.0.1:/var/local/lib/3trsh_lib/etc 1.2G 816M 329M 72% /vol/etc 127.0.0.1:/var/local/lib/3trsh_lib/lib 1.2G 816M 329M 72% /vol/lib /dev/hdc 63G 46G 15G 76% /vol/_impex
Typical symptoms of a corrupted meta volume might be:
In any case, before considering a meta volume as corrupted or in error, care should be taken to verify that there are errors about it in the messages or dmesg of the server where the controller is started, especially indications of failed fsck or similar messages
Procedure for repairing an inconsistent meta volume
The first thing that may be attempted if we suspect a corrupted controller meta volume is to mount it to a block device and run fsck against it, then restart the controller making sure it is taking the repaired meta volume.
To do this, the following steps may be followed:
3tsrv sd get
cluster
{
signature = "S20120402215634478274868435381"
}
volume boot
{
mirrors
{
mirror v-ctl-boot: server = srv1, synced = 1
mirror v-e69315e2-fff3-473c-97dc-af309466134a: server = srv2, synced = 1
}
}
volume meta
{
mirrors
{
mirror v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208: server = srv1, synced = 1
mirror v-1b0ecc0e-782c-4ddb-b8cb-b7981560fd3e: server = srv2, synced = 1
}
}
volume impex
{
mirrors
{
mirror v-ctl-impex: server = srv1, synced = 1
mirror v-f821f907-4ae0-4044-87ea-6dd39cefe9fd: server = srv2, synced = 0
}
}
server srv2: ha_role = secondary
server srv1: ha_role = primary
server lodg6srv8.lod.ca.labs.com: ha_role = reference
recovery: password = 'AppLogic01'
mkdir /var/applogic/meta_backup cd /var/applogic/meta_backup cp -p /var/applogic/volumes/vols/v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208.meta.original cp -p /var/applogic/vols/ v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 .
hosetup /dev/hoop150 v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 mdadm -assemble /dev/md150 -run /dev/hoop150 mdadm: /dev/md150 has been started with 1 drive (out of 2)
mkdir /var/applogic/meta_backup/temp mount -t ext3 /dev/md150 /var/applogic/meta_backup/temp cd /var/applogic/meta_backup/temp ls apps cluster.desc cores license lost+found rbac state users catalogs conf ha logs picture scripts stats volumes cd /var/applogic/meta_backup umount /var/applogic/meta_backup/tempç fsck /dev/md150 fsck 1.41.1 (01-Sep-2008) e2fsck 1.41.1 (01-Sep-2008) /dev/md150: clean, 1449/65536 files, 17039/262144 blocks
mdadm --stop /dev/md150 mdadm: stopped /dev/md150 hosetup -d /dev/hoop150
service heartbeat stop (in case this is running, first the secondaries, and then the primary) xm destroy controller 3tsrvctl list mounts | grep SYS mnt.srv1.SYSTEM:_sys.boot 2 /dev/md15 mnt.srv1.SYSTEM:_sys.meta 2 /dev/md16 mnt.srv1.SYSTEM:_sys.impex 2 /dev/md17 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.boot 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.meta 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.impex 3tsrv bd list | grep v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 hoop42 v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 Y 63043 hosetup -d /dev/hoop42 (no need to destroy the other hoop devices as we will reboot)
3tsrv sd get > /tmp/meta_file
vi /tmp/meta_file (in red the line that should be marked as 0, non synchronized)
cluster
{
signature = "S20120402215634478274868435381"
}
volume boot
{
mirrors
{
mirror v-ctl-boot: server = srv1, synced = 1
mirror v-e69315e2-fff3-473c-97dc-af309466134a: server = srv2, synced = 1
}
}
volume meta
{
mirrors
{
mirror v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208: server = srv1, synced = 1
mirror v-1b0ecc0e-782c-4ddb-b8cb-b7981560fd3e: server = srv2, synced = 0
}
}
volume impex
{
mirrors
{
mirror v-ctl-impex: server = srv1, synced = 1
mirror v-f821f907-4ae0-4044-87ea-6dd39cefe9fd: server = srv2, synced = 1
}
}
server srv2: ha_role = secondary
server srv1: ha_role = primary
server lodg6srv8.lod.ca.labs.com: ha_role = reference
3tsrv sd set file=/tmp/meta_file
3tsrv sd get (to verify that the change has been properly saved to the meta data)
cd /var/applogic/meta_backup cp -p v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 /var/applogic/volumes/vols/v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208
3tsrv reboot
After some time the controller will come up with the corrected meta volume
Procedure for Replacing the meta Volume
In case it is not possible to repair the meta volume using fsck and it needs to be replaced from a backup, it is also possible to do it.
Differently to the boot volume, the applogic distribution does not include a pre-formatted boot volume that may be easily replaced. So, we will have to follow these steps before we can replace the corrupted volume:
The meta volume as of version 3.X and below is formed by a single partitioned volume 1073 Mbyte in size, with a block size of 512 byte (in all 2097152 blocks). So to create it we will do:
mkdir /var/applogic/meta_backup cd /var/applogic/meta_backup dd if=/dev/zero of=meta_volume bs=512 count=2097152 ls -la meta_volume rw-rr- 1 root root 1073741824 Mar 20 08:43 meta_volume We will create one supplementary data stream which we will use to create the raid1 volume and format it and which we will not use afterwards dd if=/dev/zero of=meta_volume2 bs=512 count=2097152 And to format it subsequently we will mount it to a hoop and then create an ext3 filesystem on it: losetup -f /dev/loop1 losetup /dev/loop1 meta_volume losetup -f /dev/loop2 losetup /dev/loop2 meta_volume2 mdadm -create /dev/md150 -level=1 --raid-device=2 /dev/loop1 /dev/loop2 mkfs -t ext3 /dev/md150 mke2fs 1.41.1 (01-Sep-2008) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 38 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
mkdir /var/applogic/meta_backup/temp cd /var/applogic/meta_backup/ mount /dev/md150 temp/
From the BFC in the /opt/bfc/applogic_versions directory corresponding to your version do:
scp meta <node_id>:/var/applogic/meta_backup/temp apl-meta.tar.bz2 100% 59KB 58.7KB/s 00:00 The apl-meta.tar.bz2 contains the meta basic structure and will need to be uncompressed inside the mounted meta volume in the server where we are working: cd /var/applogic/meta_backup/temp bunzip2 apl-meta.tar.bz2 tar xvf apl-meta.tar -strip-components=1
The strip-components is just to remove the applogic trailing dir to the extracted file. After this the apl-meta.tar file may be removed from the /var/applogic/meta_backup/temp directory, and the structure will look like that of a typical meta volume:
rm apl-meta.tar rm: remove regular file `apl-meta.tar'? y ls -la total 64 drwxr-xr-x 13 root root 4096 Mar 20 10:50 . drwxr-xr-x 4 root root 4096 Mar 20 09:24 .. drwxrwxr-x 2 500 500 4096 May 16 2006 apps drwxrwxr-x 6 500 500 4096 Dec 1 2008 catalogs lrwxrwxrwx 1 root root 27 Mar 20 10:47 cluster.desc -> /etc/applogic/applogic.conf drwxr-xr-x 3 root root 4096 Sep 23 2011 conf drwxrwxr-x 2 500 500 4096 Jun 10 2006 license drwxr-xr-x 2 root root 4096 Apr 30 2010 logs drwx------ 2 root root 16384 Mar 20 10:28 lost+found drwxr-xr-x 2 root root 4096 May 28 2009 picture drwxr-xr-x 2 root root 4096 Jun 2 2010 scripts drwxr-xr-x 3 500 500 4096 May 26 2009 state drwx------ 2 500 500 4096 May 16 2006 users drwxrwxr-x 2 root root 4096 May 16 2006 volumes
We will adjust permissions and possible missing directories later. The specific directories for each grid, for instance license, or conf, or volumes, will be empty, however.
"Grid metadata backup succeeded at 2013-03-13 04:12:34"
Copy the corresponding metadata file from the BFC to the mounted new metadata file in the node:
cd /opt/bfc/grid_metadata_backup/<grid_name> scp metadata_backup_<grid_name>_03_13_2013_04_12_34_317228.tar.bz2 <node_id>:/var/applogic/meta_backup/temp metadata_backup<grid_name>03_13_2013_04_12_34 100% 3420KB 3.3MB/s 00:00
Now back to the node where we are rebuilding the volume, uncompress the metadata backup in its locationbunzip2 metadata_backup_<grid_name>_03_13_2013_04_12_34_317228.tar.bz2
bunzip2: metadata_backup<grid_name>03_13_2013_04_12_34_317228.tar.bz2: trailing garbage after EOF ignored
Ignore the messages about EOF and trailing garbage. These exist even if the backup is correct.
Now uncompress the resulting file in the mounted new metadata volume:
tar xvf metadata_backup_<grid_name>_03_13_2013_04_12_34_317228.tar -strip-components=2
The --strip-components=2 option needs to be specified because the tar file contains the reference to /var/applogic and we are already uncompressing in what will be that directory when mounted. Now the metadata backup file may be deleted, and the resulting directories inside the new metadata volume basically contain the correct structure and contents:
rm metadata_backup_<grid_id>_03_13_2013_04_12_34_317228.tar rm: remove regular file `metadata_backup<grid_name>03_13_2013_04_12_34_317228.tar'? y ls -la drwxr-xr-x 14 root root 4096 Mar 20 11:19 . drwxr-xr-x 4 root root 4096 Mar 20 09:24 .. drwxrwxr-x 95 500 500 4096 Mar 13 06:55 apps drwxrwxr-x 10 500 500 4096 May 25 2012 catalogs lrwxrwxrwx 1 root root 27 Mar 20 10:47 cluster.desc -> /etc/applogic/applogic.conf drwxr-xr-x 3 root root 4096 Mar 6 17:42 conf drwxrwxr-x 2 500 500 4096 Mar 13 04:08 license drwxr-xr-x 2 root root 4096 Apr 30 2010 logs drwx------ 2 root root 16384 Mar 20 10:28 lost+found drwxr-xr-x 2 root root 4096 May 28 2009 picture drwxr-xr-x 4 root root 4096 Mar 6 17:43 rbac drwxr-xr-x 2 root root 4096 Jun 2 2010 scripts drwxr-xr-x 3 500 500 4096 May 26 2009 state drwx------ 2 500 500 4096 Mar 13 03:22 users drwxrwxr-x 2 root root 20480 Mar 13 06:55 volumes
cd /var/applogic/meta_backup umount temp/ mdadm --stop /dev/md150 mdadm: stopped /dev/md150 losetup -d /dev/loop1 losetup -d /dev/loop2 rm meta_volume2
3tsrv sd get
cluster
{
signature = "S20120402215634478274868435381"
}
volume boot
{
mirrors
{
mirror v-ctl-boot: server = srv1, synced = 1
mirror v-e69315e2-fff3-473c-97dc-af309466134a: server = srv2, synced = 1
}
}
volume meta
{
mirrors
{
mirror v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208: server = srv1, synced = 1
mirror v-1b0ecc0e-782c-4ddb-b8cb-b7981560fd3e: server = srv2, synced = 1
}
}
volume impex
{
mirrors
{
mirror v-ctl-impex: server = srv1, synced = 1
mirror v-f821f907-4ae0-4044-87ea-6dd39cefe9fd: server = srv2, synced = 0
}
}
server srv2: ha_role = secondary
server srv1: ha_role = primary
server lodg6srv8.lod.ca.labs.com: ha_role = reference
recovery: password = 'AppLogic01'
cd /var/applogic/meta_backup
cp -p /var/applogic/volumes/vols/v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208.meta.original
service heartbeat stop (in case this is running, first the secondaries, and then the primary) xm destroy controller 3tsrvctl list mounts | grep SYS mnt.srv1.SYSTEM:_sys.boot 2 /dev/md0 mnt.srv1.SYSTEM:_sys.meta 2 /dev/md1 mnt.srv1.SYSTEM:_sys.impex 2 /dev/md2 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.boot 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.meta 3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.impex 3tsrv bd list | grep v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 hoop1 v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208 Y 63043 hosetup -d /dev/hoop1
3tsrv sd get > /tmp/meta_file
vi /tmp/meta_file (in red the line that should be marked as 0, non synchronized)
cluster
{
signature = "S20120402215634478274868435381"
}
volume boot
{
mirrors
{
mirror v-ctl-boot: server = srv1, synced = 1
mirror v-e69315e2-fff3-473c-97dc-af309466134a: server = srv2, synced = 1
}
}
volume meta
{
mirrors
{
mirror v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208: server = srv1, synced = 1
mirror v-1b0ecc0e-782c-4ddb-b8cb-b7981560fd3e: server = srv2, synced = 0
}
}
volume impex
{
mirrors
{
mirror v-ctl-impex: server = srv1, synced = 1
mirror v-f821f907-4ae0-4044-87ea-6dd39cefe9fd: server = srv2, synced = 1
}
}
server srv2: ha_role = secondary
server srv1: ha_role = primary
server lodg6srv8.lod.ca.labs.com: ha_role = reference
3tsrv sd set file=/tmp/meta_file
3tsrv sd get (to verify that the change has been properly saved to the meta data)
cd /var/applogic/meta_backup cp -p meta_volume /var/applogic/volumes/vols/v-ddbe6bb2-9cfc-4c4c-b8e0-95b1f9f99208
3tsrv reboot
After some time the controller will come up with the recovered meta volume
|
Copyright © 2012 CA.
All rights reserved.
|
|