Getting Ready › Release Notes › Hotfixes › Hotfix hf8751

Hotfix hf8751

Release date: 2013-10-14

hf8751 resolves the performance issue reported in bug SCR 8751. This bug was originally reported with the title Performance issues with megaraid_sas driver in Applogic 3.1+ though it was later determined that it is CPU-related, and not related to the Megaraid device.

The issue only occurs on relatively new CPUs. The exact range of CPU models that are affected has not been determined, but all CPUs that support Extended Page Tables (EPT) appear to be affected.

This hotfix requires a grid reboot.

Applicability:

hf8751 is an optional but highly recommended hotfix for CA AppLogic® users that want to gain improved performance on certain newer CPU architectures. The hotfix is released in two different packages, one for AppLogic 3.5 and another for AppLogic 3.7.

Details:

The hotfix addresses a bug in the Linux kernel, which causes it to waste a lot of CPU time during I/O processing when running in PV mode on XEN. The bug affects both the AppLogic control OS (XEN Dom0) and guest VMs running a Linux-based OS (DomU).

Note: To obtain the maximum benefit from applying the fix, any guest VMs (appliances) should also have an updated kernel.

This hotfix replaces the control domain (Dom0) kernel and also provides a pre-built guest kernel that can be used in appliances, if necessary.

If your virtual appliance is using a stock OS kernel, the following operating systems are affected by the bug and must have their kernel replaced in addition to installing the hotfix:

• CentOS-6
• Ubuntu 10.4

If your virtual appliance is using a stock OS kernel, the following operating systems are not affected by the bug:

• CentOS-5

  Note: CentOS-5 comes with Linux 2.6.18, and does not exhibit the excessive CPU load on I/O when running in XEN. However, this kernel performs overall slightly worse than newer Linux kernels when used in a virtual machine. If performance is important, using a new kernel can help, but note that if a new kernel is installed in place of the CentOS kernel, the automatic update from the CentOS repositories does not keep the kernel up to date.

• Ubuntu 12.4 (if updated to the latest kernel from the apt repository)

If your virtual appliance is using a custom-built kernel, or using one of the pre-built para-virtualized kernels provided with AppLogic:

• linux-2.6.18 is not affected, no action is needed (though replacing it may improve performance as described in the previous Note).
• linux-2.6.30, 2.6.32.x (longterm stable branch) and newer 2.6 kernels are affected and need to be replaced or patched.
• linux-3.2 versions 3.2.46 and later have the bug fixed. Earlier versions need to be replaced.

Other versions of Linux not already mentioned:

• Check if the kernel has the patch found in the mainline kernel archive that is listed in the GIT change log as:
  xen: correctly check for pending events when restoring irq flags

Preparing for Installation of a New Kernel in Appliances with linux-2.6

Important: When installing the new (linux-3.2) kernel into appliances that were using linux-2.6, it may be necessary to modify the boot command line and the system mount table. This is because the new kernel uses virtual disk device names that are different from those used by Linux-2.6. Complete the following steps before attempting to install the new kernel and do not reboot the appliance without completing the boot configuration and mount table changes.

1. Run the blkid command to display labels and UUIDs of all disks. Either can be used for specifying mounts, UUID is recommended.

   blkid
   

   The output will be similar to this:

   /dev/hda1: UUID="5090dfea-e6a1-4aed-b758-509b1fd6f07e" TYPE="ext3"
   

2. Ensure your operating system is configured to use plain device names by searching for the following strings in the cat /boot/grub/menu.list and cat /etc/fstab files:
   • /dev/hda1
   • /dev/sda1
3. If the boot device or mounts are specified by device name, change them to use UUID.

   For example, if the following line is included in the menu.lst file:

   kernel  /boot/vmlinuz-2.6.18-194.32.1.el5xen root=/dev/hda1 ro
   

   change it as follows (using your device UUID, not the one in this example):

   kernel  /boot/vmlinuz-2.6.18-194.32.1.el5xen root=root=UUID=5090dfea-e6a1-4aed-b758-509b1fd6f07e ro
   

4. Edit the /etc/fstab file to use UUID. For example, change the following line:

   /dev/hda1    /   ext3    defaults,noatime        0 0
   

   to:

   UUID=5090dfea-e6a1-4aed-b758-509b1fd6f07e  / ext3  defaults,noatime   0 0
   

Installing the Guest Kernel in Appliances

For new appliances being created by installing a new Linux-based operating system, you can use the hvm2pv utility to install the new kernel that is included with this hotfix.

For existing appliances, or if you are not using the hvm2pv utility, complete the following steps:

1. Log in to the appliance using a text ssh shell (for example, from the GUI interface, or using the comp ssh command in the CLI).
2. Download the new kernel and a copy of APK (even if APK is already installed) as follows:

   cd /tmp
   ctl="${SSH_CLIENT%% *}"
   

   For 64-bit appliances:

   wget http://$ctl:8080/download/domu-3.2.0.53.amd64.tar.gz
   

   For 32-bit appliances:

   wget http://$ctl:8080/download/domu-3.2.0.53.i386.tar.gz
   

   In the following command, adjust the APK version to whatever your grid has available in the download directory. Also, use the APK version that corresponds with your operating system (for example, apk-*-linux-rh for CentOS and RHEL, # -linux-ub for Ubuntu and Debian, and so on):

   wget http://$ctl:8080/download/apk-3.0.11-linux-rh.tar.gz
   

3. Unpack the archives as follows (this example uses a 32-bit CentOS appliance and apk-3.0.11):

   cd / tar -zxf tmp/domu-3.2.0.53.i386.tar.gz tar -zxf tmp/apk-3.0.11-linux-rh.tar.gz
   

4. Run the APK installation program as follows:

   tmp/apk-install
   

   This configures the new kernel to boot in XEN PV mode.