Guide To Troubleshooting Stuck Volumes

This is for internal use only since we do not want customers running the commands listed in the document.

Common scenarios caused by stuck volumes:

Is the volume really stuck?

Verify the volume repair daemon is not running and suspend if necessary: $>3t vol repair --suspend time=60
Verify the volumes are in an 'available' state: $> 3t vol list <appname> --all
- If a volume is listed as 'in_use' but the application is not running and the volume is not being managed then you can issue the following to force a umount: $> 3tctl umount sv <appname>.class.<volname> Note: user volumes will be in the form of <appname>.user.<volname>
Can you manage the volume? $> 3t vol manage <appname>:<volname>
- If this is succesful then the volume can be mounted and the filesystem is intact.
- A corrupt filesystem may still cuase this test to fail.
Can you mount the volume as a block device on the controller? $> 3tctl mount sv <appname>.class.<volname>
- Even with a corrupt filesystem the volume will still mount as a block device.
If vol manage and/or mounting the volume as a block device is successfull then this is not a stuck mount. If mounting as a block device fails continue below.

Determine were a volume is stuck:

The first place you need to check is the controller logs. '3t log list' or as maintainer: $>cat /var/log/messages | grep -v sshd (Look for messages regarding an inability to mount a volume stream)
- Any message that lists failed to mount on 127.0.0.1 is always going to be a stuck mount on the controller.
Normally the error output will list a particular node that the mount has failed on, if not there is 3 nodes you will need to check.
- 2 of them will be the nodes that host the volume streams of the app. $> 3t vol info <appname>.<volname>
- The 3rd node is were the application is attempting to start at. $> 3t comp list <appname>

Fixing controller stuck mounts:

These are the easiest to clear up as normally there should be no mounted volumes on the controller. Note: Volumes are mounted briefly during app build/start.
To list the stuck md devices: $> cat /proc/mdstat
For any md devices listed you can stop them with: $> mdadm --stop /dev/mdX

Fixing stuck mounts on grid nodes:

First place to check is /var/log/messages. Most of the time it will specify a md or hoop device that is failing to mount to.
You can also run $>3tsrv bd list --all to see if any device is unused.
A unused md device will have no hoop device attached to it and the hoop device would have no volume attached.
To stop a unused md device $> mdadm --stop /dev/mdX Note: Stopping a device that is in use can lead to data loss/corruption.
To stop a unused hoop device $> hosetup -d /dev/hoopX Note: Stopping a device that is in use can lead to data loss/corruption.