Reference Information › CA AppLogic Support Knowledge Base › Overview of Support Knowledge Base › How to Replace Corrupt Controller Boot Volume

How to Replace Corrupt Controller Boot Volume

These steps would normally be done on the primary node.

We first need to determine the grid version so we can get the correct controller boot image. Since the controller is most likely not running you can check on the BFC UI or from a grid node you can check /etc/applogic.d/applogic.env.ver file. The controller boot image will be in the Applogic distribution directory on the BFC. We want to use the files in /opt/bfc/applogic_versions/ Do no use the files in the download directory. The download directory has the 'vanilla' sources and will not have any distro hotfixes applied which could contain updates to the controller boot image. The two files we need are ctl-boot-3t.tar.bz2 and v-ctl-boot.bz2. scp or rsync both of these files to the grid node were the controller boot volume stream is located.

Quick Steps:

1. Copy ctl-boot-3t.tar.bz2 and v-ctl-boot.bz2 to the server where you want to put the stream.
2. bunzip2 v-ctl-boot.bz2.
3. Attach v-ctl-boot to a hoop device and create an md device with the hoop device.
4. Mount the md device.
5. Untar ctl-boot.3t.tar.bz2 into the mount point.
6. Unmount and cleanup the devices.
7. Backup the current controller boot volume stream.
8. Overwrite the controller boot volume stream with the newly created stream.
9. Mark only the new stream as OK.
10. Reboot the server and if it is not the controller server, also reboot the controller server.

Detailed steps taken Josh's bash history session during a case (reassembled by Shannon, so blame me if it's not perfect):

Copy ctl-boot-3t.tar.bz2 and v-ctl-boot.bz2 from the distro directory to the server where you want to put the stream.

3tsrv sd get > /root/temp

vi /root/temp - remove the synced mirror =1 on the secondary node

3tsrv sd set file=/root/temp

3tsrv sd get

ls

rm -rf v-ctl-boot

bunzip2 ctl-boot-3t.tar.bz2

ls

exit

bunzip2 v-ctl-boot.bz2

Attach v-ctl-boot to a hoop device and create an md device with the hoop device.

hosetup /dev/hoop240 v-ctl-boot

mdadm --assemble /dev/md240 --force --run /dev/hoop240

sfdisk -uB -l /dev/md240

Mount the md device. (If the offset is not 32768 then follow this doc to find the correct offset: Mounting volumes as a block device on controller )

mount -t ext3 -o offset=32768 /dev/md240 /mnt/temp/

cd /mnt/temp

Untar ctl-boot.3t.tar.bz2 into the mount point.

tar jxvfp /var/applogic/ctl-boot-3t.tar.bz2

Unmount and cleanup the devices.

umount /mnt/temp

Backup the current controller boot volume stream.

cd /var/applogic/

ls

cp v-ctl-boot /var/applogic/volumes/vols/v-ctl-boot

Overwrite the controller boot volume stream with the newly created stream.

3tsrvctl list mounts

3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.boot

3tsrv bd list | grep md1

cp v-ctl-boot /var/applogic/volumes/vols/v-ctl-boot

cat /proc/mdstat

3tsrv bd list --all | more

3tsrv sd get

Remove mounts

hosetup -d /dev/hoop240

mdadm -stop /dev/hoop240

3tsrv bd list --hoop

hosetup -d /dev/hoop0

xm list

cat /var/spool/apl-srv/srvmon

rm /var/spool/apl-srv/srvmon

3tsrvctl list mounts

3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.meta

3tsrvctl destroy mount mnt.srv1.SYSTEM:_sys.impex

3tsrv bd list --hoop

hosetup -d /dev/hoop1

3tsrvctl list mounts

3tsrv bd list --all

3tsrv bd list --all | grep hoop1

3tsrv set role=primary --recover

Reboot the server and if it is not the controller server, also reboot the controller server.