Previous Topic: Root Cause of 3THA: HA Check Failed - Other Operation Currently in ProgressNext Topic: Sample of Troubleshooting App Migration Issue


Running but Needs Attention Grid State after BFC Upgraded to 3.7

So far, we saw the following grid state issues after the customer upgraded BFC from earlier version 3.7.

Problem:

AppLogic 3.5 grid with "Running but needs attention" state with the following status:

2013-07-26 21:18:55 : Grid PGIDEV (3139) - State information is now: Running, but needs attention.: Failed updating grid server power credentials. Use clear failure to acknowledge the failure. The operation will be retried when the power credentials change again.

connecting to the grid servers OK
reading current configuration
reading controller configuration
reading server configuration
reading cfg template from 192.168.10.2
reading current configuration OK
updating configuration
updating server configuration
updating controller configuration
undefined parameter 46:jumbo_frames_enabled=
updating configuration FAILED, aborting
cleanup
closing connection to 192.168.10.254
closing connection to 192.168.10.2
closing connection to 192.168.10.3
closing connection to 192.168.10.4
closing connection to 192.168.10.5
cleanup OK
Workaround Solution:

The grid is version 3.5, but it has a property map file from 3.7 (3.7 does not have jumbo_frames_enabled setting, it has jumbo_frames_mode instead).

The correct version of the map file can be restored from the 3.5 distro file "/opt/bfc/applogic_versions/3.5.19/aldlib/lib/ctl-config.map" (rsync in archive mode should be used for this, to ensure the file date is set properly).

The location of the file that we need to replace is:

"/usr/local/applogic/bin/ctl-config.map" on the grid controller.

You should replace the file before doing anything with BFC on that grid. There should be no need to restart BFC, nor the grid.

Then, do "Clear Failure" from BFC UI, and the grid state will change back to "Running".

Star issue 21512790-1
Problem:

AppLogic 3.1 grid with "Running but needs attention" state with the following status:

2013-08-26 09:05:42 : Grid RSGRID4 (3000) - State information is now: Running, but needs attention.: Failed updating grid server power credentials. Use clear failure to acknowledge the failure. The operation will be retried when the power credentials change again.
2013-08-26 09:05:42 : Grid RSGRID4 (3000) - State information is now: Failed updating grid server power credentials. Use clear failure to acknowledge the failure. The operation will be retried when the power credentials change again.
2013-08-26 09:05:39 : Grid RSGRID4 (3000) - State information is now: Error running aldo command: ['set', 'grid=RSGRID4', 'servers=192.168.50.249:192.168.51.125:PowerAdmin_BFC:*****,192.168.50.254:192.168.51.181:PowerAdminBFC:****,192.168.50.250:192.168.51.180:PowerAdmin_BFC:*****', 'answer=yes'] - returned Traceback (most recent call last):

File "/opt/bfc/sudo/ald/wrapaldo", line 30, in <module>
aldo(sys.argv)
File "/opt/bfc/sudo/ald/wrapaldo", line 23, in aldo
p = subprocess.Popen(argv[1:])
File "/usr/lib/python2.6/subprocess.py", line 623, in _init_
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1141, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Workaround Solution:

This causes by the applogic version is corrupted. Usually we encounter this error when the file “/opt/bfc/applogic_versions/<AL version>/aldo” is not in place.

  1. Do "su – bfcadmin" in the BFC ssh console
  2. Locate the directory where BFC are downloading AppLogic (from the bfc admin). Let's say it is {applogic download directory}
  3. Locate the directory with the 3.1.14 version. So that would be {applogic download directory}/3.1.14
  4. Go into the "/opt/bfc/applogic_versions/3.1.14" directory
  5. Run: "rsync -rptgoLvc --delete {applogic download directory}/3.1.14 ."
  6. Make sure "rel-ald-2.2.4.tar.bz2" is in the "/opt/bfc/applogic_versions/3.1.14" directory.
  7. Run: "tar -jxvf rel-ald-2.2.4.tar.bz2"
  8. Then, do "Clear Failure" from BFC UI, and the grid state will change back to "Running".