Virtualization Best Practices

 Last Updated: March 15, 2013

Virtualization of distributed systems requires skills, planning and the use of appropriate tools.   The purpose of this section is to help clarify what virtualization can (and cannot) do for IT as well as to provide some best practice advice in regards to both virtualization in general and virtualization for several CA solutions.

The term virtualization can have many meanings and many applications but, within the context of this page "virtualization" is used to refer to "software/hardware solutions that support multiple operating system instances on a single hardware system." Further, although there are many virtualization solutions, our discussion is focused on server virtualization using VMware ESX/ESXi, Microsoft Hyper-V and Citrix XenServer.

Note that storage virtualization is often needed to ensure optimum performance of storage I/O constrained real and virtualized servers.  Storage virtualization, as provided via SAN solutions, as well as through various RAID configurations, is discussed here at a high level however, you should have an understanding of how to tune your vendor specific solution. 

Some solutions provide network access virtualization - and this page provides some high level tuning options for virtualized network access. Some CA solutions have components where it is considered best practice to use dedicated NICs or bandwidth if you wish to virtualize for production use. These include, but are not limited to, the CA NSM Distributed State Machine (DSM), CA ITCM Scalability Servers, and high network activity components such as the MDB. In general, production use of any application with high network activity will be impacted without dedicated NICs or a robust network virtualization solution.

All solutions provide memory and CPU virtualization - and, just as you should not over commit virtual storage within a single operating system, it is critical that the number of simultaneously active virtual machines does not exceed the real memory. The number of simultaneously active VCPUs defined should also not exceed the number of physical CPUs that are available to your virtualization solution. 

All of these resource related topics are discussed in Best Practices and vendor specific product documentation. Keep in mind that CA Best Practices are not a substitute for vendor specific tuning knowledge, skilled use of appropriate tuning tools, and practical experience.

General Best Practices

Following are excerpts from the current Virtualization Best Practices Guide. Best Practices tips for virtualizing specific CA Solutions can be found in the "Virtualizing CA Solutions" section. Appendix B of the Virtualization Best Practices document also includes a Virtualization Checklist that lists things that need to be considered while implementing a virtualized solution.

Why use Virtualization?

Resource optimization is one of the key drivers for deploying virtualization technology. Regardless of whether your resources are currently under- or over-utilized, the impact affects your business's bottom line.  Virtualization offers the potential for a reduction in total server hardware and better management of that hardware but it is not without its own caveats. There are both advantages and disadvantages to using virtualization in any environment and it is critical that you understand the reality of what virtualization can offer and reconcile that with your expectations of how it can be used in your environment.

For example, you should not assume that applications are suddenly going to require fewer resources just because they are virtualized.  On the contrary, the use of virtualization adds overhead and virtualized applications often use more resources than before. The actual amount of overhead depends on a number of factors including the type of application, which virtualization engine is being used, what kind of hardware is available, and how it will be used. Before implementing virtualization you need to make sure that you have enough storage bandwidth/space, memory, CPU, network bandwidth and other resources to handle the applications AND the virtualization overhead. When you exhaust any single resource on a server that is used for virtualization, all guest operating systems may be impacted and seen to wait for that single resource. Thus, low CPU or memory utilization alone is not an indication that additional work can be added - you must verify that all required resources have available capacity.

Common Advantages

Some common advantages offered by virtualization include:

Common Disadvantages

Some disadvantages associated with virtualization include the following:

Keys to making the most of a virtualized environment

The key to making the most of virtualization is to exploit the advantages - while considering the potential impact of the disadvantages.  Examples of situations in which virtualization can help you realize quick return on investment include:

On the other hand, you may want to avoid using virtualization in situations where the advantages are not worth the risks incurred. Application owners should understand and approve risks rather than assuming virtualization is beneficial.

Strategies for Deployment

There are many virtualization vendors to select from. Most vendors provide tips and best practices for deploying their solutions. When it comes to optimizing the return on your virtualization effort certain themes appear common across vendors:

Plan, Plan and Plan some more....(and don't forget to test first)

ALL virtualization project plans need to: 

  1. Identify Business drivers
  2. Identify risks
  3. Identify expected costs and ROI

While doing the above ensure you:

Tip: When you install a virtual environment be sure to create a document that tracks what has been installed and modified.  This document can be very useful when the environment is shared by different users. Include OS particulars, a list of installed applications and any other pertinent information and keep a copy of this document on both the host and guest OS.

Be sure to optimize your resources before employing virtualization in order to reduce the potential for resource usage bottlenecks.  CA has an extensive set of tools that can identify the best candidates for virtualization, locate/manage resource constraints, and tune virtualized environments.  PerfMon can be a useful "free" tool in this respect. It can collect a wide selection of Windows performance metrics over time, at specified intervals, and generate a log or multiple logs that can be graphically analyzed to identify problem areas in your system performance.  Tips on understanding what metrics to collect and how to correctly configure this tool to collect them can be found in the "Using PerfMon Efficiently and Effectively" document.

For more information on planning your deployment, consult the Virtualization Best Practices Guide.

Estimating Hardware Requirements

When estimating the amount of hardware resources needed for your virtual environment there are many things to keep in mind. A few of the more important considerations for planning and scheduling are as follows:

Following are some important caveats regarding hardware usage in a Virtualized environment:

Important: To avoid potentially unpredictable performance it is critical to ensure that all necessary resources - not just CPU and Memory - are available when needed. For example, even though management and monitoring tools do not typically consume a lot of bandwidth, they often require significant Network I/O. Therefore, use of a dedicated NIC for each VM responsible for these types of applications is recommended (and, often required).

For more details on this see the "Estimating Hardware Requirements" section in the Virtualization Best Practices Guide.

Virtualizing CA Solutions

CA products support and manage a broad spectrum of today's leading platforms, operating systems and applications across the IT enterprise. Click here to view a list of integrations that support virtualization. Many of our clients run applications and operating systems in virtualized environments. Although CA makes extensive use of VMware within its development and support organizations to create various Windows and Linux environments (and notes any specific issues it detects in product readme files), it is important to note that CA does not attempt to test every possible setup of its products with VMware or other virtualization tools and cannot certify specific client configurations. Here is a updated March 15, 2013 convenient summary of CA support for Virtualization.

Although the following sections provide some tips for deploying CA solutions in a virtualized environment, as well as for using CA solutions such as CA Virtual Performance Management (CA VPM, formerly, CA Advanced Systems Management or CA ASM) to manage those environments, you should familiarize yourself with the current CA policy regarding support for virtualized environments.

General Notes Regarding Support

In general, CA products are certified for use on specific operating systems (OS).  If an OS supports virtualization then it is likely that the product can be deployed in that virtualized environment. To determine if your CA solution supports deployment in a specific virtualized environment, you should consult the product documentation (including the Readme file) as well as the product's Home Page on Support Online. Specific certification configuration information may be found on the compatibility matrix pages for a product (absence of product specific certification means that the CA general statements below apply).

Some CA software support guidelines regarding VMware are provided in the SupportOnline VMware page.

Note that this support statement now covers all supported Virtualization platforms.

Virtualization Notes for CA Products

This section contains information regarding virtualization of several CA products. This is not intended an exhaustive list of CA solutions.

CA IT Client Manager / CA Client Automation / CA Server Automation

CA IT Client Manager (ITCM - formerly Unicenter Desktop and Server Management) infrastructure has been successfully virtualized for test and production implementations.  The r12 release of CA ITCM is certified on VMware ESX 3.5 (See details in Compatibility Matrix), and the r11.2 release of CA ITCM in Windows operating environments is certified on VMware ESX Server 3.0.1 and 3.5 Update 2, which are preferred for production implementations. There are some restrictions regarding the Asset Management agent, which are noted in the documentation.  Information on the individual CA ITCM subcomponents is provided as follows:

CA did extensive functionality and scalability testing of CA ITCM, the results of which are summarized here.
The certification of CA ITCM r11.2 on VMware ESX in the GIS lab used the following configuration (and CA runs production instances with a similar configuration).

VMWare ESX version 3.0.1 hosted on a high end server with the following characteristics:

VMWare ESX guest sessions running the DSM software were configured as follows:

CA Asset Portfolio Management

For CA Asset Portfolio Management support statements and compatibility matrices, check the following links:

CA Service Desk

CA Service Desk is another product which has had a number of successful implementations in virtualized environments, particularly those employing VMware ESX. It has been implemented in production environments servicing up to 35,000 end users and in service provider scenarios where a single virtualized USD implementation was used to service multiple, smaller clients.

CA Spectrum & CA eHealth

For CA Spectrum you should consult the Best Practices: Modeling Virtual Environments in CA Spectrum whitepaper.

The Support Statement for all NVM products in ANY virtualization environment can be accessed at the following link:
https://support.ca.com/phpdocs/7/5637/CA_Support_Statement_for_Running_CA_NVM_Products_in_Virtualization_Environments.pdf

CA Wily

Virtualization using VMWare's ESX Server is supported for CA Wily APM if you want to take advantage of the features that this architecture offers. With the proper configuration, you can run CA Wily APM on VMWare's ESX Server with performance similar to native Windows or Linux running on the same hardware. For more information on this see "VMware Requirements and Recommendations" in the CA Wily Application Performance Management r9.0 Sizing and Performance Guide.

When implementing CA Wily Enterprise Manager in virtualized environments it is important to consult the VMware IO Requirements for EM knowledge document.

CA Security Solutions

Compatibility information regarding the CA Secure Content Manager (formerly eTrust Secure Content Manager), CA Audit and CA Security Command Center and VMware can be found at the following link:

CA Storage Solutions

Compatibility information for CA BrightStor ARCserve Backup can be found at the following links:

CA Network and Systems Management (NSM)

NSM has several components where it is best practice to use dedicated NICs if you wish to virtualize for production use. These include the Distributed State Machine (DSM), middle tier event managers or common object repository servers with high activity, and other high network activity components such as performance cube processing and the MDB.
DSMs in particular require robust hardware for virtualization. The NSM architecture is optimized for DSMs that are electronically close to the objects being polled by the DSM. The further away the DSM is from the objects being polled the fewer objects a single DSM can poll. Example, if you have DSMs deployed out in the field with minimal network latency between the DSM and its managed objects, you should use great care in moving them into a central datacenter with virtualization. There are multiple issues: the first is the increase in network latency and the second is increase in latency caused by virtualization. Also, if you virtualize multiple DSM on the same server, you should have at least one NIC per DSM and the NICs should be confirmed to work well with your virtualization platform. It is common to observe a wide range of virtualization impact for different NICs (especially across NIC vendors but also between different models of NICs from the same vendor).
If a DSM performs well at 20k objects on real hardware, it will generally not perform as well when virtualized even after a reduction in object count. You should expect to have to reduce the number of objects being polled and/or incease the poll interval when virtualizing DSMs - and if you are moving DSMs from remote locations into a centralized datacenter you will need to carefully measure performance and expect to aggresively tune object count, polling interval and even add more DSMs.

In determining whether or not to use multi-processor VCPUs for CA NSM (as well as for any application), you need to consider which applications are best suited for multi-threaded operations.  Setting up a machine with multiple VCPUs which are unused or underutilized can degrade performance for everyone else on that box because having unused CPUs delays scheduling of other guests until the owning virtual machine is idle.

Testing has shown that, for CA NSM, it is often better to set the VCPU to a single processor for performance reasons, unless you can dedicate CPUs to CA NSM. If you want to run a CA NSM R11.1 Manager (WV/MDB) on Dual VCPU's with maximum effect, force the SQL server onto the second CPU by running the following SQL configuration query:

sp_configure 'show advanced options', 1;
RECONFIGURE;
GO
sp_configure 'affinity mask', 2;
RECONFIGURE;
GO

You should see the difference immediately in improved responsiveness of the MCC.

CA Virtual Assurance for Infrastructure Managers - Managing your Virtual Environment

There are several technical challenges inherent in using virtualization technology.   These include:

Unless the appropriate management steps are taken, you can risk growing out of touch with your resources. That is, although you can see (and touch) the actual, physical server boxes, what's on the outside (i.e., a single, physical box) often belies what's on the inside (i.e., multiple virtual machines and relationships).

CA Virtual Assurance for Infrastructure Manager (formerly CA Virtual Performance Management) is an add-on product to CA Technologies Infrastructure Managers (CA eHealth Performance Manager, CA Spectrum Infrastructure Manager, CA NSM and CA Spectrum Automation Manager) that provides centralized management across physical and virtual environments, adding much-needed virtualization visibility to confidently deploy production applications in virtual environments. It supports multi-platform, multi-vendor server technologies such as VMware ESX and Virtual Center, Solaris Zones*, IBM LPARS (P6)*, and Citrix XenServer* (solution tagged with a * will be released after the initial r12 GA release).

Marquee Features

Integrations with CA Assurance and Automation Managers:

CA eHealth Performance Manager and CA Spectrum Infrastructure Manager CA Spectrum Infrastructure Manager and CA eHealth Performance Manager now provide the ultimate integrated management experience for both physical and virtual systems via CA Virtual Assurance for Infrastructure Manager. This new capability allows CA Virtual Assurance for Infrastructure Manager to proactively poll and capture critical real-time metrics directly from VMware vCenter Servers and Sun Solaris Zones Servers used in diagnosing performance problems, fault isolation and root-cause analysis reducing downtime.

CA eHealth Performance Manager:

CA Spectrum Infrastructure Manager:

CA NSM:
CA Virtual Assurance for Infrastructure Manager extends CA NSM’s systems management capabilities to include Virtual systems.

CA Spectrum Automation Manager:
CA Virtual Assurance for Infrastructure Manager integration with CA Spectrum Automation Manager delivers comprehensive monitoring, management, and reporting across virtual, physical, and cloud environments.

Benefits

CA Virtual Assurance for Infrastructure Managers is a highly scalable solution designed for large, global IT organizations offering exceptional control, agility, and deep insight into systems, networks, databases, and applications across both physical and virtual environments. Further, modular solutions enable organizations to leverage their existing investments in CA technologies. Key benefits include:

Additional Information

General Information Links

Following are several useful links that provide general information about using Virtualized environments. Although many of these documents are vendor specific they provide generic guidelines that can be extrapolated for use in other environments.
Note: Many of the links below reference external sources.

Benchmark Tests

Information and useful links for VMware vSphere/ESX/ESXi

Documentation and other generic technical resources:

Best Practice Document provided by VMware. See also Documentation and Technical Resources referenced above.

Select Knowledgebase Articles. It is important to scan the existing knowledge base articles to verify that there aren’t any known issues with your specific environment. For example, if your application depends on a precise clock it is critical that you review the latest knowledgebase articles related to timekeeping on your OS.

Information and Useful Links for VMware Test Environments

The information in this section applies to any VMware product, including solutions such as VMware Server and VMware Workstations, which are typically not recommend for production environments.

VMware Workstation Tips & Tricks to enhance the Virtual Machine performance:

Information and useful links for Microsoft Hyper-V

Information and useful links for Citrix (XenServer)

Information and useful links related to Other Solutions and Technologies

Virtualization in the "Smart Enterprise"