Resource Pools – ESX Test

Resource pools can be used to hierarchically partition available CPU and memory resources. Each standalone host has an (invisible) root resource pool that groups the resources of that host. The root resource pool is not displayed because the resources of the host and the root resource pool are always the same. If you do not create child resource pools, only the root resource pools exist. Users can create child resource pools of the root resource pool or of any user-created child resource pool. Each child resource pool owns some of the parent’s resources and can, in turn, have a hierarchy of child resource pools to represent successively smaller units of computational capability. A resource pool can contain child resource pools, virtual machines, or both. You can therefore create a hierarchy of shared resources. The resource pools at a higher level are called parent resource pools. Resource pools and virtual machines that are at the same level are called siblings.

This test monitors the amount of the physical server’s resources that each resource pool on an vSphere/ESXi server is taking up.

Target of the test : An ESX server host

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for each resource pool configured on the ESX server host that is monitored.

Configurable parameters for the test
  1. Test period - How often should the test be executed
  2. Host - The host for which the test is to be configured.
  3. port - The port at which the specified host listens. By default, this is NULL.
  4. esx user and esx password - In order to enable the test to extract the desired metrics from a target ESX server, you need to configure the test with an ESX USER and ESX PASSWORD. The user credentials to be passed here depend upon the mechanism used by the eG agent for collecting performance statistics from the ESX server and its VMs. These monitoring methodologies and their corresponding configuration requirements have been discussed hereunder:

    • Monitoring using the web services interface of the ESX server: Starting with ESX server 3.0, a VMware ESX server offers a web service interface using which the eG agent collects metrics from the ESX server. The VMware VI SDK is used by the agent to implement the web services interface. To use this interface for monitoring, this test should be configured with an ESX USER who has “Read-only” privileges to the target ESX server. By default, the root user is authorized to execute the test. However, it is preferable that you create a new user on the target ESX host and assign the “Read-only” role to him/her. The steps for achieving this have been elaborately discussed in Creating a New User with Read-Only Privileges to the ESX Server section.

      ESX servers terminate user sessions based on timeout periods. The default timeout period is 30 mins. When you stop an agent, sessions currently in use by the agent will remain open for this timeout period until ESX times out the session. If the agent is restarted within the timeout period, it will open a new set of sessions. If you want the eG agent to close already existing sessions before it opens new sessions, then you would have to configure all the tests with the credentials of an ESX user with permissions to View and stop sessions (prior to vSphere/ESX server 4.1, this was called the View and Terminate Sessions privilege). To know how to grant this permission to an ESX user, refer to section.

    • Monitoring using the vCenter in the target environment: By default, the eG agent connects to each ESX server and collects metrics from it. While this approach scales well, it requires additional configuration for each server being monitored. For example, separate user accounts may need to be created on each server for read-only access to VM details. While monitoring large virtualized installations however, the agents can be optionally configured to monitor ESX servers using the statistics already available with different vCenter installations in the environment.

    In this case therefore, the ESX USER and ESX PASSWORD that you specify should be that of an Administrator or Virtual Machine Administrator in vCenter. However, if, owing to security constraints, you prefer not to use the credentials of such users, then, you can create a special role on vCenter with ‘Read-only’ privileges.

    Refer to Assigning the ‘Read-Only’ Role to a Local/Domain User to vCenter section of this document to know how to create a user on vCenter.

    If the ESX server for which this test is being configured had been discovered via vCenter, then the eG manager automatically populates the esx user and esx password text boxes with the vCenter user credentials using which the ESX discovery was performed.

    Like ESX servers, vCenter servers too terminate user sessions based on timeout periods. The default timeout period is 30 mins. When you stop an agent, sessions currently in use by the agent will remain open for this timeout period until vCenter times out the session. If the agent is restarted within the timeout period, it will open a new set of sessions. If you want the eG agent to close already existing sessions before it opens new sessions, then you would have to configure all the tests with the credentials of a vCenter user with permissions to View and stop sessions (prior to vCenter 4.1, this was called the View and Terminate Sessions permission). To know how to grant this permission to a user to vCenter, refer to Creating a Special Role on vCenter and Assigning the Role to a Local/Domain User section.

    When the eG agent is started/restarted, it first attempts to connect to the vCenter server and terminate all existing sessions for the user whose credentials have been provided for the tests. This is done to ensure that unnecessary sessions do not remain established in the vCenter server for the session timeout period.  Ideally, you should create a separate user account with the required credentials and use this for the test configurations. If you provide the credentials for an existing user for the test configuration, when the eG agent starts/restarts, it will close all existing sessions for this user (including sessions you may have opened using the Virtual Infrastructure client). Hence, in this case, you may notice that your VI client sessions are terminated when the eG agent starts/restarts.

  5. confirm password - Confirm the password by retyping it here.
  6. ssl - By default, the ESX server is SSL-enabled. Accordingly, the SSL flag is set to Yes by default. This indicates that the eG agent will communicate with the ESX server via HTTPS by default.

    Like the ESX sever, the vCenter is also SSL-enabled by default. If you have chosen to use the vCenter for monitoring, then you have to set the SSL flag to Yes.

  7. webport - By default, in most virtualized environments, the vSphere/ESX server and vCenter listen on port 80 (if not SSL-enabled) or on port 443 (if SSL-enabled). This implies that while monitoring an SSL-enabled vSphere/ESX server directly, the eG agent, by default, connects to port 443 of the vSphere/ESX server to pull out metrics, and while monitoring a non-SSL-enabled server, the eG agent connects to port 80. Similarly, while monitoring a vSphere/ESX server via an SSL-enabled vCenter, the eG agent connects to port 443 of vCenter to pull out the metrics, and while monitoring via a non-SSL-enabled vCenter, the eG agent connects to port 80 of vCenter. 

    Accordingly, the webport parameter is set to 80 or 443 depending upon the status of the ssl flag.  In some environments however, the default ports 80 or 443 might not apply. In such a case, against the webport parameter, you can specify the exact port at which the vSphere/ESX server or vCenter in your environment listens so that the eG agent communicates with that port.

  8. VIRTUAL CENTER - If the eG manager had discovered the target ESX server by connecting to vCenter, then the IP address of the vCenter server used for discovering this ESX server would be automatically displayed against the vIRTUAL center parameter; similarly, the esx user and esx password text boxes will be automatically populated with the vCenter user credentials, using which ESX discovery was performed.

    If this ESX server has not been discovered using vCenter, but you still want to monitor the ESX server via vCenter, then select the IP address of the vCenter host that you wish to use for monitoring the ESX server from the vIRTUAL center list. By default, this list is populated with the IP address of all vCenter hosts that were added to the eG Enterprise system at the time of discovery. Upon selection, the esx user and esx password that were pre-configured for that vCenter server will be automatically displayed against the respective text boxes.

    On the other hand, if the IP address of the vCenter server of interest to you is not available in the list, then, you can add the details of the vCenter server on-the-fly, by selecting the Other option from the vIRTUAL center list. This will invoke the add vcenter server details page. Refer to Adding the Details of a vCenter Server for VM Discovery section.

    On the other hand, if you want the eG agent to behave in the default manner -i.e., communicate with each ESX server for monitoring it - then set the VIRTUAL CENTER parameter to ‘none’. In this case, the ESX USER and ESX PASSWORD parameters can be configured with the credentials of a user who has at least ‘Read-only’ privileges to the target ESX server.

  9. list all pools - By default, resource pools are discovered by first discovering the VMs and then determining which resource pools they are a part of. According to this default behavior, the list all pools flag is set to No by default. In this case, if a resource pool does not have any VMs, such resource pools will not be discovered. Note that there is not much benefit in having a resource pool without any VMs assigned to it. If you have such “empty” resource pools that have no VMs, but you want these discovered as well, then you will have to set the list all pools flag to Yes. This will force the agent to change its discovery process - to list resource pools by parent child relationships first and then to identify which resource pools have VMs.
  10. report cluster resource pools - A VMware ESX server can be a part of a cluster or it may not.

    If a VMware ESX server is not part of a cluster, its resource pools will be discovered by the agent and metrics reported for the resource pools. These metrics will appear in the VMware vSphere ESX model of eG Enterprise.

    If a VMware ESX server is part of a cluster, its resource pools and its resources will be managed by vCenter, and these clusters and resource pools will show up in the VMware vCenter model offered by eG Enterprise. If a VMware ESX server is not part of a cluster, its resource pools will not show up in the VMware vCenter model.

    Therefore, when managing an ESX server, if that server is found to be a part of a cluster, then, by default, the agent will not report metrics for its resource pools (since this information will already be available in the VMware vCenter model). This is the default behavior and corresponds to the report cluster resource pools  flag being set to No by default. If this flag is changed to Yes, metrics for resource pools will be reported in the VMware vSphere ESX model irrespective of whether the ESX server is part of a cluster or not.

  11. DETAILED DIAGNOSIS - To make diagnosis more efficient and accurate, the eG suite embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

    The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

    • The eG manager license should allow the detailed diagnosis capability
    • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measurements made by the test
Measurement Description Measurement Unit Interpretation

Child resource pools:

Indicates the total number of child resource pools under this resource pool.

Number

 

VMs directly assigned to pool:

Indicates the number of virtual machines that were directly assigned to this resource pool.

Number

 

VMs powered on:

Indicates the number of virtual machines under this resource pool that is currently powered on.

Number

If the value of this measure is equal to the value of the VMs directly assigned to pool measure, it indicates that all VMs in the pool are in a powered-on state. If the values do not match, it indicates that one/more VMs are currently in a powered-off state.

Physical CPU utilization:

Indicates the percentage of physical CPU resources used from this pool.

Percent

A high value of this measure could indicate excessive CPU usage by the child resource pools/virtual machines configured under this resource pool. By comparing the value of this measure across resource pools, you can quickly identify that resource pool which is draining the CPU resources of the ESX host.

Physical CPU used:

Indicates the CPU usage of the resource pool in Mhz

Mhz

The percent CPU usage measure serves as an effective indicator of how resource-intensive a particular resource pool is on a specific ESX server. However, for performing capacity planning or what-if analysis, the CPU usage of a pool measured in absolute terms would be more useful.  For instance, the physical CPU usage of a pool could be 30% on a particular ESX server - this means that the resource pool is consuming 30% of the total physical CPU capacity of that ESX server. If you are planning to migrate the resource pool to another ESX server, then it would be unwise to assume that the pool will only consume 30% of CPU on the other ESX server as well, as the percentage will vary depending upon the physical CPU resources that are available to the other ESX server. The absolute measure however will remain unchanged across ESX server. Therefore, to decide which ESX server a VM/resource pool is to be moved to, and to analyze the impact of this movement on the CPU resources of the new ESX host, you would require an absolute measure of CPU usage.    

CPU reservation:

Indicates the amount of CPU resources that is guaranteed available to the resource pool.

Mhz

Typically, the Reservation setting for a resource pool specifies the minimum acceptable amount of CPU or memory - not the amount you would like to have available.

Follow the guidelines mentioned below to define the Reservation setting:

  • Do not set the Reservation too high. A very high reservation can limit the number of virtual machines you can power on in a resource pool.
  • Always leave some headroom while reserving resource. Do not commit all resources. As you move closer to fully reserving all capacity in the system, it becomes increasingly difficult to make changes to reservations and to the resource pool hierarchy without violating admission control.
  • If you expect frequent changes to the total available resources, use Shares, not Reservation, to allocate resources fairly across resource pools.

CPU reservation used:

Indicates the total amount of CPU resources that have been used to satisfy the reservation requirements of all descendants of this resource pool.

Mhz

CPU resources are typically reserved for powered-on virtual machines in a resource pool only. Virtual machines that are not currently running do not use any CPU resources.

Reserved resources are not wasted if they are not used. If the utilization is less than the reservation, the resources can be utilized by other running virtual machines.

CPU unreserved:

Indicates the total amount of resources available to satisfy a reservation for a child resource pool.

Mhz

In the undercommitted state, this value is limited by the capacity at the root node. In the overcommitted case, this could be higher since we do not perform the dynamic capacity checks.

Consider a resource pool with reservation=2GHz that is

totally idle. It has 2GHz reserved, but it is not actually using any of its reservation. In such a case:

  • Other resource pools cannot reserve these 2GHz.
  • Other resource can use these 2GHz, that is, idle CPU reservations are not wasted.

CPU allocation used:

Indicates the percentage of allocated CPU resources that are being used by this resource pool.

Percent

A high value of this measure indicates that one/more VMs in the pool are consuming the allocated CPU resources excessively.

CPU limit:

Indicates the limit beyond which this resource pool should not use CPU resources, even if there are resources available.

Mhz

If the Unlimited flag is enabled for a resource pool, then this measure will report the value Unlimited. This setting helps avoid wastage of idle resources.

Memory used:

Indicates the amount of memory used by this resource pool.

 

MB

 

Memory reservation:

Indicates the amount of memory resources that is guaranteed available to the resource pool.

MB

Typically, the Reservation setting for a resource pool specifies the minimum acceptable amount of CPU or memory - not the amount you would like to have available.

Follow the guidelines mentioned below to define the Reservation setting:

  • Do not set the Reservation too high. A very high reservation can limit the number of virtual machines you can power on in a resource pool.
  • Always leave some headroom while reserving resource. Do not commit all resources. As you move closer to fully reserving all capacity in the system, it becomes increasingly difficult to make changes to reservations and to the resource pool hierarchy without violating admission control.
  • If you expect frequent changes to the total available resources, use Shares, not Reservation, to allocate resources fairly across resource pools.

Memory reservation used:

Indicates the total amount of memory resources that have been used to satisfy the reservation requirements of all descendants of this resource pool.

MB

Memory resources are typically reserved for powered-on virtual machines in a resource pool only. Virtual machines that are not currently running do not use any memory resources.

Reserved resources are not wasted if they are not used. If the utilization is less than the reservation, the resources can be utilized by other running virtual machines.

Memory unreserved:

Indicates the total amount of resources available to satisfy a reservation for a child resource pool.

MB

In the undercommitted state, this value is limited by the capacity at the root node. In the overcommitted case, this could be higher since we do not perform the dynamic capacity checks.

If the resource pool does not use the reserved memory, then remember the following:

  • Other resource pools cannot reserve these memory resources
  • Other resource pools can use these memory resources, that is, idle memory reservations are not wasted.

Memory limit:

Indicates the limit beyond which this resource pool should not use memory resources, even if there are resources available.

MB

If the Unlimited flag is enabled for a resource pool, then this measure will report the value Unlimited. This setting helps avoid wastage of idle resources.

Active memory:

Indicates the amount of memory resources actively used from this pool.

MB

High memory consumption by a resource pool, besides eroding the physical memory resources of the host, can also impact the memory allocations to other resource pools/virtual machines; the lack of adequate memory to a VM can significantly strain its performance. 

Active memory used:

Indicates the percentage of total configured or available memory in this pool.

Percent

Zero memory:

Indicates the amount of memory that is zeroed out.

MB

 

Swapped memory:

Indicates the amount of memory that is swapped.

MB

ESX Server hosts use swapping to forcibly reclaim memory from a virtual machine when no vmmemctl driver is available because the vmmemctl driver:

  • Was never installed
  • Has been explicitly disabled
  • Is not running (for example, while the guest operating system is booting)
  • Is temporarily unable to reclaim memory quickly enough to satisfy current system demands

Standard demand-paging techniques swap pages back in when the virtual machine needs them.

Swap space must be reserved on disk for any unreserved virtual machine memory. This swap reservation is required to ensure the system is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the swap space may actually be used.

Typically, swap space usage for each VM should be low. Since access from RAM is much faster than access from physical disk, excessive usage of swap memory will slow down the performance of a VM. Watch for VMs that are seeing higher swap usage and more swap reads and writes.

Memory overhead:

Indicates the memory overhead incurred by this resource pool.

MB

ESX Server virtual machines can incur two kinds of memory overhead:

  • The additional time to access memory within a virtual machine.
  • The extra space needed by the ESX Server host for its own code and data structures, beyond the memory allocated to each virtual machine.

The ESX server’s memory virtualization ensures that the time overhead to memory accesses is minimal.

The memory space overhead, on the other hand, is composed of two other components:

  • A fixed system-wide overhead for the service console (in the case of ESX 3/3.5) and the VMkernel.
  • Additional overhead for each virtual machine

In addition, the space reserved for the virtual machine frame buffer and various virtualization data structures, also add to the memory overhead.

Addition of more virtual CPUs, allocation of more memory to the guests, and choosing to configure 32-bit guest operating systems instead of 64-bit ones, can go a long way in reducing this overhead. The ESX server also provides optimizations such as memory sharing to save up memory.

Balloon memory:

Indicates the amount of memory used by memory control.

MB

The vmmectl driver that is installed on a virtual machine, emulates an increase or decrease in memory pressure on the guest operating system; this way, it forces the guest OS to place memory pages into its local swap file. This driver differs from the VMware swap file method as it forces the operating system to determine what memory it wishes to page. Once the memory is paged locally on the guest operating system, the free physical pages of memory may be reallocated to other guests. As the ESX hosts sees that memory demand has been reduced, it will instruct vmmemctl to “deflate” the balloon and reduce pressure on the guest OS to page memory.

The maximum amount of memory that can be reclaimed from a guest may be configured by modifying the “sched.mem.maxmemctl” advanced option.

If the memory reclaimed from a guest (i.e., the value of this measure) is very low, it indicates excessive memory usage by the guest. Under such circumstances, you might want to consider allocating more memory to the guest.

Shared memory:

Indicates the amount of memory in the resource pool that is shared.

MB

 

Memory granted:

Indicates the amount of memory that is granted to this resource pool.

MB