VM Details – ESX Test

This test monitors the amount of the physical server’s resources that each guest on an ESX server is taking up. Using the metrics reported by this test, administrators can determine which virtual guest is taking up most CPU, which guest is generating the most network traffic, which guest is taking up the maximum memory utilization, which guest has the maximum disk activity, etc. Note that the amount of resources taken up by a virtual guest will be limited by the resource allocations that have been made by administrators. For example, an administrator could cap the amount of memory that a specific guest may take. Also, virtual guests can be organized into resource pools, and allocation of resources can be made at the resource-pool level. In this case, virtual guests allocated to the same resource pool contend for the resources allocated to the resource pool.

Target of the test : An ESX server host

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for every logged-in user/powered guest on the ESX server monitored.

Configurable parameters for the test

  1. Test period - How often should the test be executed
  2. Host - The host for which the test is to be configured.
  3. port - The port at which the specified host listens. By default, this is NULL.
  4. esx user and esx password - In order to enable the test to extract the desired metrics from a target ESX server, you need to configure the test with an ESX USER and ESX PASSWORD. The user credentials to be passed here depend upon the mechanism used by the eG agent for collecting performance statistics from the ESX server and its VMs. These monitoring methodologies and their corresponding configuration requirements have been discussed hereunder:

    • Monitoring using the web services interface of the ESX server: Starting with ESX server 3.0, a VMware ESX server offers a web service interface using which the eG agent collects metrics from the ESX server. The VMware VI SDK is used by the agent to implement the web services interface. To use this interface for monitoring, this test should be configured with an ESX USER who has “Read-only” privileges to the target ESX server. By default, the root user is authorized to execute the test. However, it is preferable that you create a new user on the target ESX host and assign the “Read-only” role to him/her. The steps for achieving this have been elaborately discussed in Creating a New User with Read-Only Privileges to the ESX Server section.

      ESX servers terminate user sessions based on timeout periods. The default timeout period is 30 mins. When you stop an agent, sessions currently in use by the agent will remain open for this timeout period until ESX times out the session. If the agent is restarted within the timeout period, it will open a new set of sessions. If you want the eG agent to close already existing sessions before it opens new sessions, then you would have to configure all the tests with the credentials of an ESX user with permissions to View and stop sessions (prior to vSphere/ESX server 4.1, this was called the View and Terminate Sessions privilege). To know how to grant this permission to an ESX user, refer to Section .

    • Monitoring using the vCenter in the target environment: By default, the eG agent connects to each ESX server and collects metrics from it. While this approach scales well, it requires additional configuration for each server being monitored. For example, separate user accounts may need to be created on each server for read-only access to VM details. While monitoring large virtualized installations however, the agents can be optionally configured to monitor ESX servers using the statistics already available with different vCenter installations in the environment.

    In this case therefore, the ESX USER and ESX PASSWORD that you specify should be that of an Administrator or Virtual Machine Administrator in vCenter. However, if, owing to security constraints, you prefer not to use the credentials of such users, then, you can create a special role on vCenter with ‘Read-only’ privileges.

    Refer to Assigning the ‘Read-Only’ Role to a Local/Domain User to vCenter section of this document to know how to create a user on vCenter.

    If the ESX server for which this test is being configured had been discovered via vCenter, then the eG manager automatically populates the esx user and esx password text boxes with the vCenter user credentials using which the ESX discovery was performed.

    Like ESX servers, vCenter servers too terminate user sessions based on timeout periods. The default timeout period is 30 mins. When you stop an agent, sessions currently in use by the agent will remain open for this timeout period until vCenter times out the session. If the agent is restarted within the timeout period, it will open a new set of sessions. If you want the eG agent to close already existing sessions before it opens new sessions, then you would have to configure all the tests with the credentials of a vCenter user with permissions to View and stop sessions (prior to vCenter 4.1, this was called the View and Terminate Sessions permission). To know how to grant this permission to a user to vCenter, refer to Creating a Special Role on vCenter and Assigning the Role to a Local/Domain User section.

    When the eG agent is started/restarted, it first attempts to connect to the vCenter server and terminate all existing sessions for the user whose credentials have been provided for the tests. This is done to ensure that unnecessary sessions do not remain established in the vCenter server for the session timeout period.  Ideally, you should create a separate user account with the required credentials and use this for the test configurations. If you provide the credentials for an existing user for the test configuration, when the eG agent starts/restarts, it will close all existing sessions for this user (including sessions you may have opened using the Virtual Infrastructure client). Hence, in this case, you may notice that your VI client sessions are terminated when the eG agent starts/restarts.

  5. confirm password - Confirm the password by retyping it here.
  6. ssl - By default, the ESX server is SSL-enabled. Accordingly, the SSL flag is set to Yes by default. This indicates that the eG agent will communicate with the ESX server via HTTPS by default.

    Like the ESX sever, the vCenter is also SSL-enabled by default. If you have chosen to use the vCenter for monitoring, then you have to set the SSL flag to Yes.

  7. webport - By default, in most virtualized environments, the vSphere/ESX server and vCenter listen on port 80 (if not SSL-enabled) or on port 443 (if SSL-enabled). This implies that while monitoring an SSL-enabled vSphere/ESX server directly, the eG agent, by default, connects to port 443 of the vSphere/ESX server to pull out metrics, and while monitoring a non-SSL-enabled server, the eG agent connects to port 80. Similarly, while monitoring a vSphere/ESX server via an SSL-enabled vCenter, the eG agent connects to port 443 of vCenter to pull out the metrics, and while monitoring via a non-SSL-enabled vCenter, the eG agent connects to port 80 of vCenter. 

    Accordingly, the webport parameter is set to 80 or 443 depending upon the status of the ssl flag.  In some environments however, the default ports 80 or 443 might not apply. In such a case, against the webport parameter, you can specify the exact port at which the vSphere/ESX server or vCenter in your environment listens so that the eG agent communicates with that port.

  8. exclude vms - Administrators of some virtualized environments may not want to monitor some of their less-critical VMs - for instance, VM templates - both from ‘outside’ and from ‘inside’. The eG agent in this case can be configured to completely exclude such VMs from its monitoring purview. To achieve this, provide a comma-separated list of VMs to be excluded from monitoring in the exclude vms text box. Instead of VMs, VM name patterns can also be provided here in a comma-separated list. For example, your exclude vms specification can be: *xp,*lin*,win*,vista. Here, the * (asterisk) is used to denote leading and trailing spaces (as the case may be). By default, this parameter is set to none indicating that the eG agent obtains the inside and outside views of all VMs on a virtual host by default. By providing a comma-separated list of VMs/VM name patterns in the exclude vms text box, you can make sure the eG agent stops collecting ‘inside’ and ‘outside’ view metrics for a configured set of VMs.
  9. ignore vms inside view - Administrators of some high security VMware environments might not have permissions to internally monitor one/more VMs. The eG agent can be configured to not obtain the inside view of such ‘inaccessible’ VMs using the ignore vms inside view parameter. Against this parameter, you can provide a comma-separated list of VM names, or VM name patterns, for which the inside view need not be obtained. For instance, your ignore vms inside view specification can be: *xp,*lin*,win*,vista. Here, the * (asterisk) is used to denote leading and trailing spaces (as the case may be). By default, this parameter is set to none indicating that the eG agent obtains the inside view of all VMs on an ESX host by default.

    Note:

    While performing VM discovery, the eG agent will not discover the operating system of the VMs configured in the ignore vms inside view text box.

  10. ignore winnt - By default, the eG agent does not support the inside view for VMs executing on Windows NT operating systems. Accordingly, the ignore winnt flag is set to Yes by default.
  11. VIRTUAL CENTER - If the eG manager had discovered the target ESX server by connecting to vCenter, then the IP address of the vCenter server used for discovering this ESX server would be automatically displayed against the vIRTUAL center parameter; similarly, the esx user and esx password text boxes will be automatically populated with the vCenter user credentials, using which ESX discovery was performed.

    If this ESX server has not been discovered using vCenter, but you still want to monitor the ESX server via vCenter, then select the IP address of the vCenter host that you wish to use for monitoring the ESX server from the vIRTUAL center list. By default, this list is populated with the IP address of all vCenter hosts that were added to the eG Enterprise system at the time of discovery. Upon selection, the esx user and esx password that were pre-configured for that vCenter server will be automatically displayed against the respective text boxes.

    On the other hand, if the IP address of the vCenter server of interest to you is not available in the list, then, you can add the details of the vCenter server on-the-fly, by selecting the Other option from the vIRTUAL center list. This will invoke the add vcenter server details page. Refer to Adding the Details of a vCenter Server for Guest Discovery

    On the other hand, if you want the eG agent to behave in the default manner -i.e., communicate with each ESX server for monitoring it - then set the VIRTUAL CENTER parameter to ‘none’. In this case, the ESX USER and ESX PASSWORD parameters can be configured with the credentials of a user who has at least ‘Read-only’ privileges to the target ESX server.

  12. inside view using - By default, this test communicates with every VM remotely and extracts “inside view” metrics. Therefore, by default, the inside view using flag is set to Remote connection to VM (Windows).

    Typically, to establish this remote connection with Windows VMs in particular, eG Enterprise requires that the eG agent be configured with domain administrator privileges. In high-security environments, where the IT staff might have reservations about exposing the credentials of their domain administrators, this approach to extracting “inside view” metrics might not be preferred. In such environments therefore, eG Enterprise provides administrators the option to deploy a piece of software called the eG VM Agent (Windows) on every Windows VM; this VM agent allows the eG agent to collect “inside view” metrics from the Windows VMs without domain administrator rights. Refer to Configuring the eG Agent to Collect Current Hardware Status Metrics section for more details on the eG VM Agent. To ensure that the “inside view” of Windows VMs is obtained using the eG VM Agent, set the inside view using flag to eG VM Agent (Windows). Once this is done, you can set the domain, admin user, and admin password parameters to none.

  13. domain, admin user, admin password, and confirm password - By default, this test connects to each virtual guest remotely and attempts to collect “inside view” metrics. In order to obtain a remote connection, the test must be configured with user privileges that allow remote communication with the virtual guests. The first step towards this is to specify the DOMAIN within which the virtual guests reside. The admin user and admin password will change according to the domain specification. Discussed below are the different values that the domain parameter can take, and how they impact the admin user and admin password specifications:

    • If the VMs belong to a single domainIf the guests belong to a specific domain, then specify the name of that domain against the domain parameter. In this case, any administrative user in that domain will have remote access to all the virtual guests. Therefore, an administrator account in the given domain can be provided in the ADMIN USER field and the corresponding password in the ADMIN PASSWORD field. Confirm the password by retyping it in the CONFIRM PASSWORD text box.
    • If the guests do not belong to any domain (as in the case of Linux/Solaris guests) :  In this case, specify “none” in the DOMAIN field, and specify a local administrator account name in the ADMIN USER below.

      Prior to this, you need to ensure that the same local administrator account is available or is explicitly created on each of the virtual machines to be monitored. Then, proceed to provide the password of the ADMIN USER against ADMIN PASSWORD, and confirm the password by retyping it in the CONFIRM PASSWORD text box.

      If key-based authentication is implemented between the eG agent and the SSH daemon of a Linux guest, then, in the admin user text box, enter the name of the user whose <user_home_dir> (on that Linux guest) contains a .ssh directory with the public key file named authorized_keys. The admin password in this case will be the passphrase of the public key; the default public key file that is bundled with the eG agent takes the password eginnovations. Specify this as the admin password if you are using the default private/public key pair that is bundled with the eG agent to implement key-based authentication. On the other hand, if you are generating a new public/private key pair for this purpose, then use the passphrase that you provide while generating the pair. For the detailed procedure on Implementing Key-based Authentication refer to Troubleshooting the Failure of the eG Remote Agent to Connect to or Report Measures for Linux Guests section.

    • If the guests belong to different domains - In this case, you might want to provide multiple domain names. If this is done, then, to access the guests in every configured domain, the test should be configured with the required user privileges; this implies that along with multiple DOMAIN names, multiple ADMIN USER names and ADMIN PASSWORDs would also have to be provided. To help administrators provide these user details quickly and easily, the eG administrative interface embeds a special configuration page. To access this page, simply click on the Click here hyperlink that appears just above the parameters of this test in the test configuration page. To know how to use the special page, refer to VM Details – ESX Test section.
    • If the inside view using flag is set to ‘eG VM Agent (Windows)’ - In this case, the inside view can be obtained without domain administrator privileges. Therefore, set the domain, admin user, and admin password parameters to none.
  14. REPORT BY USER - While monitoring a VMware ESX server, the REPORT BY USER flag is set to No by default, indicating that by default, the guest operating systems on the ESX server are identified using the hostname specified in the operating system. On the other hand, while monitoring VMware Desktop environments, this flag is set to Yes by default; this implies that in case of VDI servers, by default, the guests will be identified using the login of the user who is accessing the guest OS. In other words, in VDI environments, this test will, by default, report measures for every username_on_virtualmachinename.
  15. REPORT POWERED OS - This flag becomes relevant only if thereport by user flagis set to ‘Yes’.

    If the report powered os flag is set to Yes (which is the default setting), then this test will report measures for even those VMs that do not have any users logged in currently. Such guests will be identified by their virtualmachine name and not by the username_on_virtualmachinename. On the other hand, if the report powered os flag is set to No, then this test will not report measures for those VMs to which no users are logged in currently.      

  16. REPORT POWERED ON - You can set the REPORT POWERED ON status to YES, so that the test reports an additional measure, Is VM powered on?, revealing whether a guest OS is currently running or not. The default status of this flag is set to Yes for a VMware ESX server. However, the default status of this flag is No in the case of virtual desktop infrastructures (VDI); this is because, in such environments, the virtual desktops will be in the powered-off state most of the time.
  17. AGGREGATE USER SESSIONS -This flag is closely related to the REPORT BY USER flag. Since the REPORT BY USER flag is set to No by default for a VMware ESX server, this test will, by default, ignore the status of the AGGREGATE USER SESSIONS flag while monitoring that server. In case of the VDI model on the other hand, the REPORT BY USER flag is set to Yes by default. Therefore, the status of the AGGREGATE USER SESSIONS flag gains significance in the case of the VDI server. By default, the AGGREGATE USER SESSIONS flag is set to No. This implies that if a single user is currently logged into multiple guests, then this test, by default, will report a set of measures for every username on guestname. On the other hand, if the status of this flag is changed to Yes, then, this test will report a set of (aggregated) measures for every distinct user to the virtual desktop environment. In other words, this test will report measures that are aggregated across all the currently active sessions for a user, spanning multiple VMs.
  18. ALERT FOR STATE CHANGE - By default, the ALERT FOR STATE CHANGE flag is set to False, indicating that alerts will not be generated when the state of a VM changes from Powered on to Powered off. If this flag is set to True, then when the eG agents starts collecting metrics, alerts will be generated only for that particular VM whose state is changed from Powered on to Powered off.

Measurements made by the test

Measurement Description Measurement Unit Interpretation

Current sessions:

This measure is relevant only for monitoring of virtual desktops (i.e., for VMware VDI or VMware vSphere VDI servers). When reporting metrics for specific users, this metric indicates the number of sessions that each user has currently logged into; this measure will be available only if the test reports measures per currently logged in user.

Number

This is a good indicator of how busy the user is. The detailed diagnosis of this measure, if enabled, reveals the guests to which the user is currently logged on to.

VM power-on state:

Whether the virtual machine is currently running on the ESX server host or not.

Boolean

The table below displays the States that can be reported by this measure, and their numeric equivalents:

State Value
Off 0
On 1
Suspended 2
Unknown 3
Standby 4
Shuttingdown 5
Resetting 6
Stuck 7

Note:

By default, this measure reports one of the States listed in the table above. The graph of this measure however will represent the VM status using the numeric equivalents - ‘0’ to ‘7’.

Physical CPU utilization:

Indicates the percentage of physical CPU used by the guest.

Percent

A high value for this measure indicates a virtual machine that is using a lot of the processor - possibly because one or more processes on this VM are taking a lot of CPU.  By correlating the values of these metrics with the VM CPU ready metric will indicate if the VM is getting sufficient processor time or not. If the values of these metrics are high and VM CPU ready value is low, this is indicative of a VM that is getting enough processor time.

Physical CPU used:

Indicates the CPU used by this guest (in Mhz)

Mhz

The percent CPU usage measure  serves as an effective indicator of how resource-intensive a particular VM is on a specific ESX server. However, for performing capacity planning or what-if analysis, the CPU usage of a VM measured in absolute terms would be more useful.  For instance, the physical CPU usage of a VM could be 30% on a particular ESX server - this means that the VM is consuming 30% of the total physical CPU capacity of that ESX server. If you are planning to migrate the VM to another ESX server, then it would be unwise to assume that the VM will only consume 30% of CPU on the other ESX server as well, as the percentage will vary depending upon the physical CPU resources that are available to the other ESX server. The absolute measure however, will remain unchanged across ESX servers. Therefore, to decide which ESX server the VM is to be moved to, and to analyze the impact of this movement on the CPU resources of the new ESX host, you would require an absolute measure of CPU usage.    

System usage of physical CPU:

Indicates the percentage of time this guest spent in the ESX server VMKernel to process interrupts and to  perform other  system  activities.

Percent

An unusually high value indicates a problem and may be due to too many system-level tasks executing simultaneously.

VM CPU waits:

Indicates the percentage of time the guest spent in the wait state.

Percent

While the VM CPU ready metric denotes the time when the VM is waiting for CPU to execute, the busy wait state represents the time when the VM is waiting for some event to happen before it is ready to execute.

VM CPU extra:

Indicates the percentage of extra CPU used up by the virtual guests.

Percent

Ideally, the value of this measure should be very low. A high value indicates that one/more processes on the VM are consuming more CPU than necessary. Typically, screen savers, animations, and X servers all consume extra CPU, which can affect

performance of other virtual machines and consolidation ratios. The non-trivial consumption of CPU by idling virtual machines can also have an adverse impact on Dynamic Resource Scheduling (DRS) decisions. You might want to disable screen savers and Window animations in the virtual machine to avoid the erosion of CPU resources. On Linux, if using an X server is not required, disable it.

VM CPU ready:

Indicates the percentage of time the guest was ready to run (i.e., it had instructions to execute) but was not able to because of processor contention.

Percentage

This metric should typically be low - generally 5% or less. The more time a VM spends waiting to run, the more lag time there is in responsiveness within the VM.

CPU guaranteed to VM:

Indicates the percentage of CPU guaranteed on the guest.

Percent

In some situations, system administrators want to know that a certain amount of memory for a virtual machine comes directly from the physical resources of the ESX

Server machine. Similarly, the administrator might want to guarantee that a certain virtual machine always receives a higher percentage of the physical resources than other virtual machines. To ensure this, you can reserve physical resources of the host for specific virtual machines. Reservation specifies the guaranteed reservation for a virtual machine. The server allows you to power on a virtual machine only if the CPU and memory reservation is available. The server guarantees that amount even when the physical server is heavily loaded.

Physical CPU throttled:

Indicates the percentage of time the ESX host deliberately did not run the  VM because that would violate the VM’s limit setting.

Percent

A Limit refers to the maximum CPU the host can make available to this VM.

Even though the VM is ready to run, if it is prevented from running owing to a probable limit violation, then the value of this measure will not be included in the value of the VM CPU ready measure.

A high value for this metric indicates that the VM has been trying to get additional CPU resources but has not been able to do so because the CPU usage for this VM or its resource pool (if it is a part of a resource pool) has reached the allocated limit.

Memory overhead:

Indicates the memory overhead incurred by the virtual machine.

MB

ESX Server virtual machines can incur two kinds of memory overhead:

  • The additional time to access memory within a virtual machine.
  • The extra space needed by the ESX Server host for its own code and data structures, beyond the memory allocated to each virtual machine.

The ESX server’s memory virtualization ensures that the time overhead to memory accesses is minimal.

The memory space overhead, on the other hand, is composed of two other components:

  • A fixed system-wide overhead for the service console (in the case of ESX 3/3.5) and the VMkernel.
  • Additional overhead for each virtual machine

In addition, the space reserved for the virtual machine frame buffer and various virtualization data structures, also add to the memory overhead.

Addition of more virtual CPUs, allocation of more memory to the guests, and choosing to configure 32-bit guest operating systems instead of 64-bit ones, can go a long way in reducing this overhead. The ESX server also provides optimizations such as memory sharing to save up memory.

Memory swapped out:

Indicates the rate at which memory is being swapped to disk by the ESX Server.

MB

ESX Server hosts use swapping to forcibly reclaim memory from a virtual machine when no vmmemctl driver is available because the vmmemctl driver:

  • Was never installed
  • Has been explicitly disabled
  • Is not running (for example, while the guest operating system is booting)
  • Is temporarily unable to reclaim memory quickly enough to satisfy current system demands

Standard demand-paging techniques swap pages back in when the virtual machine needs them.

Swap space must be reserved on disk for any unreserved virtual machine memory. This swap reservation is required to ensure the system is able to preserve virtual machine memory under any circumstances. In practice, only a small fraction of the swap space may actually be used.

Typically, swap space usage for each VM should be low. Since access from RAM is much faster than access from physical disk, excessive usage of swap memory will slow down the performance of a VM. Watch for VMs that are seeing higher swap usage and more swap reads and writes.

Current swap memory:

Indicates the amount of memory that is currently swapped.

MB

Memory swapped in:

Indicates the amount of memory that is being swapped in by the ESX Server from the disk for the guest.

MB

Swap memory target:

Indicates the amount of memory that can be swapped.

MB

Physical memory consumed:

Indicates the amount of physical memory consumed by the guest operating system.

MB

High memory consumption over long periods can deplete the free memory on the host, causing prolonged delays in the execution of the virtual machines.  Comparison of the active memory usage across guests indicates the guest(s) that could be causing a memory bottleneck on the host.

Shared memory:

Indicates the current amount of shared guest operating system memory.

MB

 

Balloon memory:

Indicates the total amount of physical memory currently reclaimed from this guest using the vmmemctl modules.

MB

The vmmectl driver that is installed on a virtual machine, emulates an increase or decrease in
memory pressure on the guest operating system; this way, it forces the guest OS to place
memory pages into its local swap file. This driver differs from the
VMware swap file method as it forces the operating system to determine
what memory it wishes to page. Once the memory is paged
locally on the guest operating system, the free physical pages of memory may be reallocated to other guests. As the ESX hosts sees that
memory demand has been reduced, it will instruct vmmemctl to
“deflate” the balloon and reduce pressure on the guest OS to page
memory.

The maximum amount of memory that can be reclaimed from a guest may be configured by modifying the “sched.mem.maxmemctl” advanced option.

If the memory reclaimed from a guest (i.e., the value of this measure) is very low, it indicates excessive memory usage by the guest. Under such circumstances, you might want to consider allocating more memory to the guest.

Balloon memory target:

Indicates the total amount of physical memory the ESX Server host attempts to reclaim using the vmmemctl module.

 

 

Zero memory:

Indicates the amount of memory that is zeroed out.

MB

The “Memory Zero” amount will fluctuate as memory is over allocated. ESX will zero out the VM’s memory to use with other VM’s.

Memory granted:

Indicates the amount of memory that the Vmkernel has granted to the virtual machine.

MB

 

Active memory:

Indicates the amount of memory that is actively used by the guest.

MB

 

Memory usage:

Indicates the percentage of active memory that is currently in use.

Percent

High memory consumption over long periods can deplete the free memory on a VM, causing prolonged delays in the execution of the applications hosted by the virtual machine. 

Configured memory:

Indicates the amount of configured memory.

MB

 

Disk reads:

Indicates the rate at which read commands were issued to the storage adapters of the guest.

Reads/Sec

 

Disk writes:

Indicates the rate at which write commands were issued to the storage adapters of the guest.

Writes/Sec

 

Issued disk commands:

Indicates the number of commands issued per second.

Commands/Sec

This measure is a good indicator of the level of activity on the storage adapters of the guest.

Disk commands aborted:

Indicates the rate at which commands aborted.

Aborts/Sec

A high rate is indicative of a problem condition that requires investigation.

Data writes to disk:

Indicates the rate at which data was written to the disk.

MB/Sec

 

Data reads from disk:

Indicates the rate at which data was read from the disk.

MB/Sec

 

Disk capacity:

Indicates the total disk capacity of the virtual machine.

MB

One of the key problems faced by VMware administrators is VM sprawl. Since VMs are easy to create and deploy, many a time an administrator might be faced with scenarios where many VMs are created on an ESX server, but very few are actively used. A VM, whether powered on or off, consumes disk space on an ESX server. When the ESX server hosting runs low on disk space, administrators might want to know which VM is taking up maximum disk space.

This measure reveals the disk capacity of a VM, regardless of its on/off state. A quick comparison of the capacity across VMs can enable administrators to accurately identify the VM that is taking up maximum disk space.

Network packets transmitted:

Indicates the number of packets the guest transmitted per second.

Packets/Sec

 

Network data transmitted:

Indicates the amount of data the guest transmitted per second.

Mbps

Comparing the data transmitted across all the virtual guests provides an indicator of the guest that is generating most out-bound network traffic.

Network packets received:

Indicates the number of packets received per second by the guest.

Packets/Sec

 

Network data received:

Indicates the rate at which data was received by the guest.

Mbps

Comparing the data received across all the virtual guests provides an indicator of the guest that has the most in-bound network traffic.

Virtual CPU utilization:

Indicates the percentage of allocated CPU resources that this VM is currently using.

Percent

Comparing the value of this measure across VMs will enable you to accurately identify the VMs on which CPU-intensive applications are executing. 

Memory limit:

Indicates the upper bound for memory resources that can be allocated to this VM.

MB

A server can allocate more than the reservation to a virtual machine, but never allocates more than the limit, even if there is unutilized memory on the system.

By default, the memory limit is set to unlimited. When the memory limit is unlimited, the amount of memory configured for the virtual machine when it was created becomes its effective limit in most cases.

In most cases, it is not necessary to specify a limit. There are benefits and drawbacks. Assigning a limit is useful if you start with a small number of virtual machines and want to manage user expectations. Performance deteriorates as you add more virtual machines. You can simulate having fewer resources available by specifying a limit.

On the other hand, you might waste idle resources if you specify a limit. The system does not allow virtual machines to use more resources than the limit, even when the system is underutilized and idle resources are available. Specify the limit only if you have good reasons for doing so.

Is memory configured correctly?

Indicates whether the memory is configured correctly or not for this VM.

 

If the configured memory is greater than the memory limit reported by the Memory limited measure, then this measure will report the value No.

If the configured memory is equal to the memory limit reported by the Memory limited measure or, if the memory limit is set to unlimited, then this measure will report the value Yes.

The numeric values that correspond to the measure values discussed above are listed in the table below:

Numeric Value Measure Value
0 Yes
1 No

Note:

By default, this measure reports the values Yes or No only to indicate whether the memory is configured correctly or not. The graph of this measure however, will represent the memory state using the numeric equivalents - 0 or 1.

Disk throughput:

Indicates the rate of data reads and writes to the disk of this VM.

MB/Sec

This is the sum of the Data reads from disk and Data writes to disk measures. This is a good indicator of the level of disk I/O activity on a VM. A very low value or a consistent decrease in the value of this measure could indicate a current/potential I/O processing bottleneck. You can compare the value of this measure across VMs, to know which VM is the slowest in terms of the speed with which it processes I/O requests. 

Total network I/O operations:

Indicates the rate at which data was received and sent across the network by this VM.

Mbps

This is the sum of the Network data transmitted and Network data received measures. This is a good measure of the level of network I/O activity on a VM. A very low value or a consistent decrease in the value of this measure could indicate a current/potential network I/O processing bottleneck. You can compare the value of this measure across VMs, to know which VM is the slowest in terms of the speed with which it processes network I/O requests. 

Connection status:

 

Indicates the current status of the vCenter server’s connection to this virtual machine.  

 

 

 

The table below lists the values that this measure can take and briefly describes each value:

Measure Value Description

Connected

The vCenter server has access to the virtual machine.

Disconnected

The vCenter server is currently disconnected from the virtual machine, since its host is disconnected.

Inaccessible

One or more of the virtual machine configuration files are inaccessible. For example, this can be due to transient disk failures. In this case, no configuration can be returned for a virtual machine. 

Invalid

The virtual machine configuration format is invalid. Thus, it is accessible on disk, but corrupted in a way that does not allow the server to read the content. In this case, no configuration can be returned for a virtual machine.

Orphaned

The virtual machine is no longer registered on the host it is associated with. For example, a virtual machine that is unregistered or deleted directly on a host managed by vCenter shows up in this state.

The numeric values that correspond to these measure values are as follows:

Numeric Value Measure Value
0

Orphaned

1

Connected

2

Disconnected

3

Inaccessible

4

Invalid

Note:

By default, this measure reports the Measure Values to indicate the connection status. In the graph of this measure however, status will be represented using the numeric equivalents - 0 through 4.

Time in the COSTOP state:

Indicates the percentage of time that this virtual machine was ready to execute commands but is waiting for the availability of multiple vCPUs as the virtual machine is configured to use multiple vCPUs.

Percent

VMware schedules all the vCPUs in a VM at the same time. If all the allocated vCPUs are not available at the same time, then the VM will be in a state of "co-stop" until the host can co-schedule all vCPUs. In its simplest form co-stop indicates the amount time after the first vCPU is available until the remaining vCPUs are available for the VM to run.

If a virtual machine is unresponsive and the value of this measure is proportionally high when compared to the value of the VM CPU ready measure. it may indicate that the VM has limited CPU resources to simultaneously co-schedule all vCPUs in the virtual machines. Sizing VMs to use the least number of vCPU’s possible minimizes the time needed for co-stop waits.

If the value of this measure is low, then any performance problems should be attributed to other issues and not to the co-scheduling of the vCPU.

VM CPU overlap:

Indicates the percentage of time the virtual machine was interrupted to perform system services on behalf of that virtual machine or other virtual machines.

Percent

If the value of the Virtual CPU utilization measure is unusually high for a VM, you may want to check the value of the VM CPU overlap measure to figure out whether it is owing to system services performed by the virtual machine.

Time in RUN state:

Indicates the percentage of time the virtual machine is consuming CPU resources.

Percent

This value represents the percentage of absolute time the virtual machine was running on the system.

If the value is near ‘zero’, it could mean that the VM is idle, blocked on an operation, or is not scheduled due to resource contention.

If it is near 100%, it means that all vCPUs in the virtual machine are busy. This is an indicator that the guest operating system may be stuck in a operational loop.

Compressed memory consumed by VM

Indicates the amount of compressed memory that this VM has consumed.

MB

vSphere provides a memory compression cache to improve virtual machine performance when you use memory overcommitment. Memory compression is enabled by default. When a host’s memory becomes overcommitted, vSphere compresses virtual pages and stores them in memory.

Because accessing compressed memory is faster than accessing memory that is swapped to disk, memory compression in vSphere allows you to overcommit memory without significantly hindering performance. When a virtual page needs to be swapped, vSphere first attempts to compress the page. Pages that can be compressed to 2 KB or smaller are stored in the virtual machine’s compression cache, increasing the capacity of the host.

The value of this measure indicates the amount of memory consumed by the compressed pages in the compression cache of a VM.

Ideally, this value should be much less than the maximum size upto which the compression cache has been configured to grow. If the value of this measure is close or equal to the maximum size configuration of the compression cache, then it means that the compression cache is filling up rapidly, and will soon run out of memory. When this happens, a lot of replaced compressed pages will be decompressed and swapped out to accommodate more compressed pages in the cache. Any following swapins of those pages will hurt VM performance. This can happen if the maximum size of the compression cache is set too low.

At the same time, a very large compression cache may also waste VM memory and unnecessarily create host memory pressure, especially when most compressed pages would not be touched in the future.

By prudently setting the maximum size of the compression cache, you can ensure optimal usage of the cache and can avoid the eventualities discussed above.

Distributed CPU entitlement

Indicates the amount of CPU resource, that the VM is entitled to, as calculated by DRS (Distributed Resource Scheduler).

MHz

This measure is reported only for a VM managed by DRS.

This measure will report a value only for those VMs that are managed by DRS.

VMware DRS dynamically balances computing capacity across a collection of hardware resources aggregated into logical resource pools, continuously monitoring utilization across resource pools and intelligently allocating available resources among the virtual machines based on pre-defined rules that reflect business needs and changing priorities. When a virtual machine experiences an increased load, VMware DRS automatically allocates additional resources by redistributing virtual machines among the physical servers in the resource pool.

By comparing the value of this measure across VMs, you can identify those VMs that DRS has marked as CPU-intensive, and has hence allocated more CPU resources.

Distributed memory entitlement

Indicates the amount of memory, in MB, that this VM is entitled to, as calculated by DRS (Distributed Resource Scheduler).

MB

This measure is reported only for a VM managed by DRS.

VMware DRS dynamically balances computing capacity across a collection a logical resource pools, continuously monitoring utilization across resource pools and intelligently allocating available resources among the virtual machines based on pre-defined rules that reflect business needs and changing priorities. When a virtual machine experiences an increased load, VMware DRS automatically allocates additional resources by redistributing virtual machines among the physical servers in the resource pool.

By comparing the value of this measure across VMs, you can identify those VMs that DRS has marked as memory-intensive, and has hence allocated more memory resources.

Private memory

Indicates the portion of memory, that is granted to this VM from non-shared host memory.

MB

Private memory is the amount of memory that is actually stored in the physical memory of the vSphere host, for a virtual machine. In other words, this is the memory that is not shared, but is backed by the host memory.

Ideally, the value of this measure should be low. Any increase in the value of this measure for a VM, will automatically increase the value of the Physical memory consumed measure of that VM. This is because:

Private memory + Memory overhead = Physical memory consumed

Static CPU entitlement

Indicates the static CPU resource entitlement for this VM.

MHz

This value is calculated based on this virtual machine’s resource reservations, shares and limit, and doesn’t take into account current usage. This is the worst case CPU allocation for this virtual machine – i.e., the amount of CPU resource this virtual machine would receive if all virtual machines running in the cluster went to maximum consumption.

Static memory entitlement

Indicates the static memory resource entitlement for this VM.

MHz

This value is calculated based on this virtual machine’s resource reservations, shares and limit, and doesn’t take into account current usage. This is the worst case memory allocation for this virtual machine – i.e., the amount of memory this virtual machine would receive if all virtual machines running in the cluster went to maximum consumption.

VM CPU idle

Indicates the percentage of time this VM did not use the CPU.

Percent

 

VM CPU shares

Indicates the number of CPU shares allocated.

Number

This is used to determine resource allocation in case of a resource contention.

This value is only set if CPU allocation level is set to custom.

VM CPU limit

Indicates the limit beyond which this VM will not use CPU, even if resources are available.

MHz

If the value of this measure is -1, then it means that there is no fixed limit on CPU usage.

VM CPU reservations

Indicates the amount of resource that is guaranteed available to this virtual machine.

MHz

 

VM CPU latency

Indicates the percent of time the virtual machine is unable to run because it is contending for access to the physical CPU(s).

Percent

If the value of this measure is close to 100% for a VM, it implies that that VM's CPU requirement is more than its allocation and its entitlement. You may want to resize the VMs to avoid such a serious contention.

VM CPU entitlement

Indicates the CPU resources devoted by the ESXi scheduler.

MHz

The ESXi scheduler implements a proportional share–based algorithm which allocates CPU resources to VMs based on their resource specifications. Users can specify the CPU allocation using shares, reservations, and limits. Then, the scheduler dynamically evaluates the priority based on the CPU consumption and the entitlement. The user controls the entitlement, but the consumption depends on many factors including scheduling, workload behavior, and system load. Also, the degree of the difference between two entitlements dictates how much CPU time should be allocated.

Maximum limited

Indicates the time this virtual machine is ready to run, but is not running because it has reached its maximum CPU limit setting.

Msecs

A low value is desired for this measure. If the value of this measure is very high for a VM, you may want to increase the maximum CPU limit setting of that VM.

VM CPU demand

Indicates the amount of CPU resources this VM would use if there were no CPU contention or CPU limit.

MHz

By observing the variations to this measure over time, you will be able to judge how much CPU resources a VM really requires.

Memory latency

Indicates the time this virtual machine is waiting to access swapped or compressed memory.

Percent

If the waiting time of a VM is very high, it can cause the performance of that VM to suffer.

Memory entitlement

Indicates the amount of host physical memory this virtual machine is entitled to, as determined by the ESX scheduler.

MB

 

Memory shares

Indicates the number of memory shares allocated to this VM.

Number

This value is used to determine resource allocation in case of resource contention.

This value is only set if memory allocation level is set to custom.

Memory reservations

Indicates the amount of memory that is guaranteed available to this VM.

MB

 

Swap in rate from host cache

Indicates the rate at which memory is being swapped from host cache into active memory.

Mbps

Ideally, the value of this measure should be high.

Swap out rate to host cache

Indicates the rate at which memory is being swapped from this VM's active memory to host cache.

Mbps

When there is severe memory pressure and the hypervisor needs to swap memory pages to disk it will swap out to the host cache on the SSD drive instead.

If the memory pages are swapped out to the host cache at a high rate - i.e., if the value of this measure is consistently high - check the amount of free physical memory on the host. A free memory value of 6% or less indicates that the host requires more memory resources.

Zipped memory

Indicates the total compressed physical memory.

MB

 

Memory saved by zipping

Indicates the amount of memory saved due to compression.

MB

A high value is desired for this measure.

Network usage

Indicates the sum of data transmitted and received across all virtual NIC instances connected to this virtual machine.

Mbps

Compare the value of this measure across VMs to know which VM is consuming the maximum bandwidth.

VM CPU swap wait

Indicates the percentage of CPU time that this VM spent waiting for a swap-in.

Percent

If the value of this measure is very high, it means that the VM was blocked for too long waiting for an I/O operation. This in turn is indicative of an I/O bottleneck on the VM.

VM CPU demand-entitlement ratio

Indicates the ratio of CPU demand to CPU entitlement for this VM.

Percent

A low value is desired for this measure. A high value indicates that the VM requires more CPU than it is entitled to receive. In such a situation, you may want to consider increasing the CPU entitlement of the VM.

Packets dropped during transmission

Indicates the number of packets that this VM dropped during transmission since the last measurement period.

Number

Ideally, the value of this measure should be 0. If packet loss is abnormally high, it could indicate a bad network connection.

Packets dropped during reception

Indicates the number of packets that this VM dropped during reception since the last measurement period.

Number

Ideally, the value of this measure should be 0. If packet loss is abnormally high, it could indicate a bad network connection.

Broadcast packets received

Indicates the rate at which this VM received broadcast packets.

Packets/Sec

Broadcast is the term used to describe communication where a piece of information is sent from one point to all other points. In this case there is just one sender, but the information is sent to all connected receivers.

Broadcast transmission is supported on most LANs (e.g. Ethernet), and may be used to send the same message to all computers on the LAN (e.g. the address resolution protocol (arp) uses this to send an address resolution query to all computers on a LAN). Network layer protocols (such as IPv4) also support a form of broadcast that allows the same packet to be sent to every system in a logical network (in IPv4 this consists of the IP network ID and an all 1's host number).

 

Broadcast packets transmitted

Indicates the rate at which this VM transmitted broadcast packets.

Packets/Sec

Multicast packets received

Indicates the rate at which this VM received broadcast packets.

Packets/Sec

Multicast is the term used to describe communication where a piece of information is sent from one or more points to a set of other points. In this case there is may be one or more senders, and the information is distributed to a set of receivers (theer may be no receivers, or any other number of receivers).

Multicasting is the networking technique of delivering the same packet simultaneously to a group of client

Multicast packets transmitted

Indicates the rate at which this VM transmitted broadcast packets.

Packets/Sec

Decompression rate

Indicates the rate at which this VM decompressed memory.

Mbps

 

Compression rate

Indicates the rate of memory compression for this VM.

Mbps

 

Host cache used for swapping

Indicates the space this VM used for caching swapped pages in the host cache.

MB

Datastores that are created on solid state drives (SSD) can be used to allocate space for host cache. The host reserves a certain amount of space for swapping to host cache.

The host cache is made up of files on a low-latency disk that ESXi uses as a write back cache for virtual machine swap files. The cache is shared by all virtual machines running on the host.

When there is severe memory pressure and the hypervisor needs to swap memory pages to disk it will swap to the host cache on the SSD drive instead.

If the value of this measure rises consistently, it indicates that the VM is swapping many memory pages to the host cache.

Also, If the value of this measure is equal to the space allocated to the host cache, it means that the memory pages swapped by that VM have completely filled the host cache. Such a VM can be marked as a memory-starved VM.

Under such circumstances, these memory pages will need to be copied to the regular .vswp file. This is not a recommended practice as it will decrease performance for your VMs as these pages more than likely at some point will need to be swapped in. To avoid this therefore, you may want to allocate more host cache space to the VM. You may also want to consider allocating more memory to that VM, so that the VM does not have to resort to swapping memory (to host cache).

Reserved overhead

Indicates the host physical memory reserved for use as the virtualization overhead for this virtual machine

MB

Overhead memory includes space reserved for the virtual machine frame buffer and various virtualization data structures, such as shadow page tables. Overhead memory depends on the number of virtual CPUs and the configured memory for the guest operating system.

Active write

Indicates the amount of memory actively being written to by the virtual machine.

MB

Compare the value of this measure across VMs to know which VM is writing to memory most actively.

Highest latency

Indicates the highest latency value across all disks used by the virtual machine.

Secs

Compare the value of this measure across VMs to identify the VM with the most latent disk.