Page 1
ESXi 4.0 vCenter Server 4.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs. EN-000107-02...
Page 2
VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
Using DRS Rules 53 Adding Hosts to a Cluster 55 Adding Virtual Machines to a Cluster 56 Remove Hosts from a Cluster 56 Removing Virtual Machines from a Cluster 57 DRS Cluster Validity 58 Managing Power Resources 62 VMware, Inc.
Page 4
Using NUMA Systems with ESX/ESXi 73 What is NUMA? 73 How ESX/ESXi NUMA Scheduling Works 74 VMware NUMA Optimization Algorithms and Settings 75 Resource Management in NUMA Architectures 76 Specifying NUMA Controls 77 Performance Monitoring Utilities: resxtop and esxtop 81...
This table provides the update history of the vSphere Resource Management Guide. Revision Description EN-000107-02 Included a point in “Multicore Processors,” on page 19 section. EN-000107-01 Removed references to CPU.MachineClearThreshold since this advanced CPU attribute is not available through the vSphere Client. EN-000107-00 Initial release. VMware, Inc.
It’s also essential for anyone who wants to understand and use resource pools, clusters, DRS, or VMware DPM. This manual assumes you have a working knowledge of VMware ESX and VMware ESXi and of vCenter Server.
Page 8
Customers with appropriate support contracts should use telephone support for the fastest response on priority 1 issues. Go to http://www.vmware.com/support/phone_support.html. To find out how VMware support offerings can help meet your business needs, Support Offerings go to http://www.vmware.com/support/services. VMware Education Services courses offer extensive hands-on labs, case study...
® A cluster is a group of hosts. You can create a cluster using VMware vCenter Server, and add multiple hosts to the cluster. vCenter Server manages these hosts’ resources jointly: the cluster owns all of the CPU and memory of all hosts.
Ensure that a certain amount of memory for a virtual machine is provided by the physical memory of the ESX/ESXi machine. Guarantee that a particular virtual machine is always allocated a higher percentage of the physical resources than other virtual machines. Set an upper bound on the resources that can be allocated to a virtual machine. VMware, Inc.
Page 11
Server or ESX/ESXi allows you to power on a virtual machine only if there are enough unreserved resources to satisfy the reservation of the virtual machine. The server guarantees that amount even when the physical server is heavily loaded. The reservation is expressed in concrete units (megahertz or megabytes). VMware, Inc.
Page 12
The following example illustrates how you can change resource allocation settings to improve virtual machine performance. Assume that on an ESX/ESXi host, you have created two new virtual machines—one each for your QA (VM- QA) and Marketing (VM-Marketing) departments. VMware, Inc.
The Resource Allocation tab is available when you select a cluster from the inventory panel. The Resource Allocation tab displays information about the CPU and memory resources in the cluster. CPU Section The following information about CPU resource allocation is shown: VMware, Inc.
Page 14
Number of megabytes (MB) not reserved. Reservations for the root resource pool of a cluster that is enabled for VMware HA might be larger than the sum of the explicitly-used resources in the cluster. These reservations not only reflect the reservations for the running virtual machines and the hierarchically-contained (child) resource pools in the cluster, but also the reservations needed to support VMware HA failover.
Page 15
Amount of memory backed by host memory and not being shared. Shared Amount of memory being shared. Swapped Amount of memory reclaimed by swapping. Ballooned Amount of memory reclaimed by ballooning. Unaccessed Amount of memory never referenced by the guest. Active Amount of memory recently accessed. VMware, Inc.
This extra memory commitment is included in the admission control calculation. When the VMware DPM feature is enabled, hosts might be placed in standby mode (that is, powered off) to reduce power consumption. The unreserved resources provided by these hosts are considered available for admission control.
Programs with a significant privileged code component, such as system calls, traps, or page table updates can run slower in the virtualized environment. VMware, Inc.
Virtualization and Processor-Specific Behavior Although VMware software virtualizes the CPU, the virtual machine detects the specific model of the processor on which it is running.
Page 19
Intel and AMD have each developed processors which combine two or more processor cores into a single integrated circuit (often called a package or socket). VMware uses the term socket to describe a single package which can have one or more processor cores with one or more logical processors in each core.
Page 20
The following processors support hyperthreading and have two threads per core. Processors based on the Intel Xeon 5500 processor microarchitecture. Intel Pentium 4 (HT-enabled) Intel Pentium EE 840 (HT-enabled) VMware, Inc.
Page 21
The VMware scheduler properly accounts for this halt time, and charges a virtual machine running with the full resources of a core more than a virtual machine running on a half core. This approach to processor management ensures that the server does not violate any of the standard ESX/ESXi resource allocation rules.
Page 22
(up to 90 percent) for an application sharing a core with the problematic code. In those cases, the ESX/ESXi host quarantines the virtual CPU running this code and places its virtual machine in the none or internal mode, as appropriate. VMware, Inc.
Page 23
CPUs). For the best performance, when you use manual affinity settings, VMware recommends that you include at least one additional physical CPU in the affinity setting to allow at least one of the virtual machine's threads to be scheduled at the same time as its virtual CPUs (for example, a uniprocessor virtual machine with affinity to at least two CPUs or a two-way SMP virtual machine with affinity to at least three CPUs).
Page 24
The VMkernel optimizes each CPU's frequency to match demand in order to dynamic improve power efficiency but not affect performance. When CPU demand increases, this policy setting ensures that CPU frequencies also increase. VMware, Inc.
RAM that is allocated to the virtual machine, which depends on the resource settings (shares, reservation, limit) explained below. VMware, Inc.
Page 26
With memory sharing, a workload consisting of multiple virtual machines often consumes less memory than it would when running on physical machines. As a result, the system can efficiently support higher levels of overcommitment. VMware, Inc.
The VMM for each virtual machine maintains a mapping from the guest operating system's physical memory pages to the physical memory pages on the underlying machine. (VMware refers to the underlying host physical pages as “machine” pages and the guest operating system’s physical pages as “physical”...
Use the attributes and special features available through the vSphere Client. The vSphere Client GUI allows you to connect to an ESX/ESXi host or a vCenter Server system. Use advanced settings. Use the vSphere SDK for scripted memory allocation. VMware, Inc.
Page 29
ESX/ESXi virtual machines can incur two kinds of memory overhead. The additional time to access memory within a virtual machine. The extra space needed by the ESX/ESXi host for its own code and data structures, beyond the memory allocated to each virtual machine. VMware, Inc.
Page 30
When memory is overcommitted, each virtual machine is allocated an amount of memory somewhere between what is specified by Reservation and what is specified by Limit. The amount of memory granted to a virtual machine above its reservation usually varies with the current memory load. VMware, Inc.
Page 31
When memory is tight, the guest operating system determines which pages to reclaim and, if necessary, swaps them to its own virtual disk. See Figure 3-2. VMware, Inc.
Page 32
Standard demand-paging techniques swap pages back in when the virtual machine needs them. For optimum performance, ESX/ESXi hosts use the ballooning approach (implemented by the driver) whenever possible. Swapping is a reliable mechanism of last resort that a host uses only when vmmemctl necessary to reclaim memory. VMware, Inc.
Page 33
SAN. However, it can lead to a slight degradation in performance for VMware VMotion because pages swapped to a local swap file on the source host must be transferred across the network to the destination host.
Page 34
ESX/ESXi memory sharing runs as a background activity that scans for sharing opportunities over time. The amount of memory saved varies over time. For a fairly constant workload, the amount generally increases slowly until all sharing opportunities are exploited. VMware, Inc.
Page 35
The number of blocks differs between the two levels due to memory sharing and so Memory Granted and Memory Consumed differ. This is not problematic and shows that memory is being saved through sharing or other reclamation techniques. VMware, Inc.
Page 36
The memory metrics that measure guest physical memory and machine memory might appear contradictory. In fact, they are measuring different aspects of a virtual machine's memory usage. By understanding the differences between these metrics, you can better utilize them to diagnose performance issues. VMware, Inc.
“Why Use Resource Pools?,” on page 38 “Create Resource Pools,” on page 39 “Add Virtual Machines to a Resource Pool,” on page 40 “Removing Virtual Machines from a Resource Pool,” on page 41 “Resource Pool Admission Control,” on page 41 VMware, Inc.
(the default). Limit Upper limit for the amount of CPU or memory the host makes available to this resource pool. Default is Unlimited. To specify a limit, deselect the Unlimitedcheck box. VMware, Inc.
Before you power on a virtual machine or create a resource pool, check the CPU Unreserved and memory Unreserved fields in the resource pool’s Resource Allocation tab to determine whether sufficient resources are available. How Unreserved CPU and memory are computed and whether actions are performed depends on the Reservation Type. VMware, Inc.
Page 42
No local resources are available for VM-K2, so it borrows resources from the parent resource pool, RP- MOM. RP-MOM has 6GHz minus 1GHz (reserved by the virtual machine) minus 2GHz (reserved by RP- KID), which leaves 3GHz unreserved. With 3GHz available, you can power on the 2GHz virtual machine. VMware, Inc.
Page 43
2GHz reserved by RP-KID). As a result, you cannot power on VM-K2, which requires a 2GHz reservation. Figure 4-4. Admission Control with Expandable Resource Pools: Power-On Prevented 6GHz RP-MOM VM-M1, 1GHz VM-M2, 2GHz 2GHz RP-KID VM-K1, 2GHz VM-K2, 2GHz VMware, Inc.
“Admission Control and Initial Placement,” on page 46. Power management— When the VMware Distributed Power Management feature is enabled, DRS compares cluster- and host-level capacity to the demands of the cluster’s virtual machines, including recent historical demand. It places (or recommends placing) hosts in standby power mode if sufficient excess capacity is found or powering on hosts if capacity is needed.
(that is, those on standalone hosts or in non-DRS clusters) are included, vCenter Server attempts to power them on automatically. If these power ons are successful, they are listed under the Started Power-Ons tab. Any virtual machines that fail to power on are listed under the Failed Power-Ons tab. VMware, Inc.
Instead, the Summary page indicates that migration recommendations are available and the DRS Recommendations page displays recommendations for changes that make the most efficient use of resources across the cluster. VMware, Inc.
Page 48
Client. A higher load imbalance leads to higher-priority migration recommendations. For more information about this metric and how a recommendation priority level is calculated, see the VMware Knowledge Base article "Calculating the priority level of a VMware DRS migration recommendation."...
In some cases, processor vendors have introduced significant architectural changes within the same processor family (such as 64-bit extensions and SSE3). VMware identifies these exceptions if it cannot guarantee successful migration with VMotion. vCenter Server provides features that help ensure that virtual machines migrated with VMotion meet processor compatibility requirements.
If the hosts are not in the VMotion network, DRS can still make initial placement recommendations. To be configured for VMotion, each host in the cluster must meet the following requirements: The virtual machine configuration file for ESX/ESXi hosts must reside on a VMware Virtual Machine File System (VMFS).
Right-click and select an automation mode. Click OK. Other VMware products or features, such as VMware vApp and VMware Fault Tolerance, might override the automation levels of virtual machines in a DRS cluster. Refer to the product-specific documentation for details.
When a cluster’s settings and resources have been defined, you should ensure that it is and remains a valid cluster. You can also use a valid DRS cluster to manage power resources and interoperate with VMware HA. This chapter includes the following topics: “Using DRS Rules,”...
Page 54
Resource Management Guide Right-Click and select Edit Settings. In the left panel under VMware DRS select Rules. Click Add. In the Virtual Machine Rule dialog box, name the rule. Select one of the options from the pop-up menu: Keep Virtual Machines Together One virtual machine cannot be part of more than one such rule.
Enter the host name, user name, and password, and click Next. View the summary information and click Next. Select what to do with the host’s virtual machines and resource pools. Put this host’s virtual machines in the cluster’s root resource pool VMware, Inc.
DRS does not recommend (or perform, in fully automated mode) any virtual machine migrations off of a host entering maintenance or standby mode if the VMware HA failover level would be violated after the host enters the requested mode.
A valid cluster has enough resources to meet all reservations and to support all running virtual machines. Figure 6-1 shows an example of a valid cluster with fixed resource pools and how its CPU and memory resources are computed. VMware, Inc.
Page 59
RP3 was created with a reservation of 3GHz. One virtual machine with 3GHz is powered on. No resources for powering on additional virtual machines are available. Figure 6-2 shows an example of a valid cluster with some resource pools (RP1 and RP3) using reservation type Expandable. VMware, Inc.
Page 60
A cluster typically turns yellow when cluster capacity is suddenly reduced, for example, when a host in the cluster becomes unavailable. VMware recommends that you leave adequate additional cluster resources to avoid your cluster turning yellow.
Page 61
You can resolve a red DRS cluster problem either by powering off one or more virtual machines, moving virtual machines to parts of the tree that have sufficient resources, or editing the resource pool settings in the red part. Adding resources typically helps only when you are in the yellow state. VMware, Inc.
If sufficient excess capacity is found, VMware DPM places one or more hosts in standby mode and powers them off after migrating their virtual machines to other hosts. Conversely, when capacity is deemed to be inadequate, DRS brings hosts out of standby mode (powers them on) and migrates virtual machines, using VMotion, to them.
Page 63
LAN (WOL). Each protocol requires its own hardware support and configuration. If a host does not support any of these protocols it cannot be put into standby mode by VMware DPM. If a host supports multiple protocols, they are used in the following order: IPMI, iLO, WOL.
Page 64
VMware guidelines. You must perform these steps before enabling VMware DPM for a cluster for the first time or on any host that is being added to a cluster that is using VMware DPM.
Page 65
Host-Level Overrides When you enable VMware DPM in a DRS cluster, by default all hosts in the cluster inherit its VMware DPM automation level. You can override this default for an individual host by selecting the host Host Options page of the cluster's Settings dialog box and clicking its Power Management setting.
Page 66
You can use event-based alarms in vCenter Server to monitor VMware DPM. The most serious potential error you face when using VMware DPM is the failure of a host to exit standby mode when its capacity is needed by the DRS cluster. You can monitor for instances when this error occurs by using the preconfigured Exit Standby Error alarm in vCenter Server.
You can access a cluster’s Summary tab from the inventory panel of the vSphere Client. The General, VMware DRS, and VMware DRS Resource Distribution sections of this tab display useful information about the configuration and operation of your cluster. The following sections describe the fields that appear in those sections.
VMware DRS Resource Distribution Chart The VMware DRS Resource Distribution chart displays CPU and memory utilization information. Open this chart by clicking the View Resource Distribution Chart link on the Summary tab for a VMware DRS cluster. CPU Utilization CPU utilization is displayed on a per-virtual machine basis, grouped by host.
Migration Automation Level Automation level for DRS virtual machine migration recommendations. Fully Automated, Partially Automated, or Manual. Power Management Automation Automation level for VMware DPM recommendations. Off, Manual, or Automatic. Level Migration Threshold Priority level (or higher) of DRS recommendations to apply.
Page 70
(significant improvement) to priority five (slight). Prior to ESX/ESXi 4.0, recommendations received a star rating (1 to 5 stars) instead of a priority level. The higher the star rating, the more desirable the move. See the VMware knowledge base article http://kb.vmware.com/kb/1007485 for information on priority level calculation.
Page 71
You can customize the display of recent actions using the Contains text box. Select the search criteria (DRS Actions, Time) from the drop-down box next to the text box and enter a relevant text string. VMware, Inc.
ESX/ESXi supports memory access optimization for Intel and AMD Opteron processors in server architectures that support NUMA (non-uniform memory access). After you understand how ESX/ESXi NUMA scheduling is performed and how the VMware NUMA algorithms work, you can specify NUMA controls to optimize the performance of your virtual machines.
The optimizations work seamlessly regardless of the type of guest operating system. ESX/ESXi provides NUMA support even to virtual machines that do not support NUMA hardware, such as Windows NT 4.0. As a result, you can take advantage of new hardware even with legacy operating systems. VMware, Inc.
Chapter 8 Using NUMA Systems with ESX/ESXi VMware NUMA Optimization Algorithms and Settings This section describes the algorithms and settings used by ESX/ESXi to maximize application performance while still maintaining resource guarantees. Home Nodes and Initial Placement When a virtual machine is powered on, ESX/ESXi assigns it a home node. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well.
NUMA nodes. This increases the amount of sharing and de-duplication, which reduces overall memory consumption at the expense of memory locality. In memory-constrained environments, such as VMware View deployments, many similar virtual machines present an opportunity for de-duplication, and page sharing across NUMA nodes could be very beneficial.
The server should allocate memory only on the specified node. Memory Affinity If you set both options before a virtual machine starts, the virtual machine runs only on the selected node and all of its memory is allocated locally. VMware, Inc.
Page 78
In the vSphere Client inventory panel, select the virtual machine and select Edit Settings. Select Options and click Advanced. Click the Configuration Parameters button. In the vSphere Client, turn on CPU affinity for processors 4, 5, 6, and 7. VMware, Inc.
Page 79
Click the Configuration Parameters button. In the vSphere Client, set memory affinity for the NUMA node to 1. Completing these two tasks ensures that the virtual machine runs only on NUMA node 1 and, when possible, allocates memory from the same node. VMware, Inc.
CLI command. resxtop Before you can use any vSphere CLI commands, you must either download and install a vSphere CLI package or deploy the vSphere Management Assistant (vMA) to your ESX/ESXi host or vCenter Server system. VMware, Inc.
The port number to connect to on the remote server. The default port is 443, and unless this is portnumber changed on the server, this option is not needed. (resxtop only) The user name to be authenticated when connecting to the remote host. The remote server prompts username you for a password, as well (resxtop only). VMware, Inc.
Erases and redraws the current panel. f or F Displays a panel for adding or removing statistics columns (fields) to or from the current panel. o or O Displays a panel for changing the order of statistics columns on the current panel. VMware, Inc.
Page 84
PCPU UTIL(%) represents the percentage of real time that the PCPU was not idle (raw PCPU utilization) and it displays the percentage CPU utilization per PCPU, and the percentage CPU utilization averaged over all PCPUs. PCPU UTIL(%) might differ from PCPU USED(%) due to power management technologies or hyperthreading. VMware, Inc.
Page 85
Percentage of time a resource pool or world spends waiting for the ESX/ESXi VMkernel to swap memory. The %SWPWT (swap wait) time is included in the %WAIT time. EVENT COUNTS/s Set of CPU statistics made up of per second event rates. These statistics are for VMware internal use only. CPU ALLOC Set of CPU statistics made up of the following CPU allocation configuration parameters.
%CSTP Percentage of time a resource pool spends in a ready, co-deschedule state. (Note: you might see this statistic displayed, but it is intended for VMware use only). You can change the display using single-key commands as discussed in Table A-5.
Page 87
Displays the ESX/ESXi page-sharing statistics. All numbers are in megabytes. Amount of physical memory that is being shared. shared Amount of machine memory that is common across worlds. common Amount of machine memory that is saved because of page saving sharing. VMware, Inc.
Page 88
%ACTVN Percentage of guest physical memory that is being referenced by the guest. This is an estimation. (You might see this statistic displayed, but it is intended for VMware use only.) MCTL? Memory balloon driver is installed or not. N means no, Y means yes.
OVHDUW (MB) Current space overhead for a user world. (You might see this statistic displayed, but it is intended for VMware use only.) GST_NDx (MB) Guest memory allocated for a resource pool on NUMA node x. This statistic is applicable on NUMA systems only.
Page 90
PAECP/s The number of PAE copies per second. SPLTCMD/s The number of split commands per second. SPLTCP/s The number of split copies per second. Table A-9 displays the interactive commands you can use with the storage adapter panel. VMware, Inc.
Number of shares. This statistic is applicable only to worlds. BLKSZ Block size in bytes. NUMBLKS Number of blocks of the device. DQLEN Storage device queue depth. This is the maximum number of ESX/ESXi VMkernel active commands that the device is configured to support. VMware, Inc.
Page 92
Number of split commands per second. This statistic is applicable only to paths. SPLTCP/s Number of split copies per second. This statistic is applicable only to paths. Table A-11 displays the interactive commands you can use with the storage device panel. VMware, Inc.
The number is valid only if the corresponding world is expanded to devices. This is a per device maximum for the world. ACTV Number of commands in the ESX/ESXi VMkernel that are currently active. This number is applicable only to worlds and devices. VMware, Inc.
Page 94
You are prompted to enter the world ID. Display virtual machine instances only. Sort by READS/s column. Sort by WRITES/s column. Sort by MBREAD/s column. Sort by MBWRTN/s column. VMware, Inc.
Sorts by Mb Tx column. Sorts by Mb Rx column. Sorts by Packets Tx column. Sorts by Packets Rx column. Sorts by PORT-ID column. This is the default sort order. Changes the displayed length of the DNAME column. VMware, Inc.
) does not accept interactive commands. In batch mode, the utility runs until resxtop esxtop it produces the number of iterations requested (see command-line option , below, for more details), or until you end the process by pressing Ctrl+c. VMware, Inc.
You do not have to run replay mode on the ESX service console. Replay mode can be run to produce output in the same style as batch mode (see the command-line option , below). VMware, Inc.
Page 98
Specifies the delay between panel updates. The default is five seconds. The minimum is two seconds. If a delay of less than two seconds is specified, the delay is set to two seconds. Number of iterations esxtop updates the display this number of times and then exits. VMware, Inc.
“Set Advanced Virtual Machine Attributes,” on page 101 Set Advanced Host Attributes You can set advanced attributes for a host. VMware recommends that only advanced users set advanced host attributes. In most cases, the AUTION default settings produce the optimum result. Procedure In the vSphere Client inventory panel, select the host to customize.
Page 100
If you enable this option, all guest memory is backed with large pages in machines that use nested page tables (for example, AMD Barcelona). If NPT is not available, only some portion of guest memory is backed with large pages. VMware, Inc.
Set Advanced Virtual Machine Attributes You can set advanced attributes for a virtual machine. Procedure Select the virtual machine in the vSphere Client inventory panel, and select Edit Settings from the right- click menu. Click Options and click Advanced > General. VMware, Inc.
Page 102
VMFS directory that contains its configuration file. This directory must remain on a host that is accessible to the virtual machine. If you move the virtual machine (or any clones created from it), you might need to reset this attribute. VMware, Inc.