Statestore Daemon TCMalloc Test
TCMalloc dynamically allocates and deallocates memory to the Statestore Daemon. TCMalloc can be categorized into three components - TCMalloc Front-end, TCMalloc Middle-end and TCMalloc Backend.
The front-end handles a request for memory of a particular size. The front-end has a cache of memory that it can use for allocation or to hold free memory. This cache is only accessible by a single thread at a time, so it does not require any locks, hence most allocations and deallocations are fast.
The middle-end is responsible for providing memory to the front-end and returning memory to the back-end. The middle-end comprises the Transfer cache and the Central free list. Although these are often referred to as singular, there is one transfer cache and one central free list per size-class. These caches are each protected by a mutex lock - so there is a serialization cost to accessing them.
The back-end of TCMalloc has three jobs:
-
It manages large chunks of unused memory.
-
It is responsible for fetching memory from the OS when there is no suitably sized memory available to fulfill an allocation request.
-
It is responsible for returning unneeded memory back to the OS.
This test help the administrators to check whether/not this TCMalloc is managing the memory allocations to the Statestore Daemon properly.
Target of the test : Apache Impala
Agent deploying the test : An internal/remote agent
Outputs of the test : One set of results for the target Statestore Daemon server.
Parameter | Description |
---|---|
Test period |
How often should the test be executed. |
Host |
The IP address of the target server that is being monitored. |
Port |
The port number through which the Apache Impala communicates. The default port is 25000. |
Impalad URL |
Each daemon server has a different endpoint URL. You need to configure the eG agent with the endpoint URL of each daemon, so that the agent can access the URL and pull metrics on the health of the target daemon servers. Specify such URL in the Impalad URL text box . To know how to configure the eG agent with the API Endpoint URLs, refer to Configuring the eG agent with the API Endpoint URLs topic in Pre-requisites for Monitoring Apache Impala. |
Impalad Authorization type |
To access daemon server, you need to configure Digest authorization. Digest authorization is a method used to verify the identity of a user requesting access to API Endpoint URL of the daemon server. Digest authorization uses a combination of a username, password, and a unique message digest (a type of encrypted code). Specify such Digest authorization value in the Impalad Authorization type text box considering the below criteria:
If multiple Impalad URLs are mentioned in the Impalad URL text box, then specify a comma-separated list of authorization type for each URL. For example, if there are two Impalad URLs, one URL without authorization and the other URL protected with authorization (username/password authorization), then specify Impalad Authorization type as: no_auth,digest_auth in the Impalad Authorization type text box. By default, Impalad Authorization type value is no_auth. |
Impalad User, Impalad Password and Confirm Password |
Once you enable web server access to the eG agent, make sure that 'password authentication' is also enabled. This is needed, so that the eG agent is able to access the built-in web server for each daemon in a safe, secure manner. Configure the credentials of such a user against Impalad User and Password text boxes. Confirm the password by retyping it in the Confirm Password text box. To know how to configure the eG agent with 'secure' access to the built-in web server, refer to Configuring the eG agent with 'secure' access to the built-in web server topic in Pre-requisites for Monitoring Apache Impala. By default, Impalad user parameter is set to none.
|
Statestored URL |
Each daemon server has a different endpoint URL. You need to configure the eG agent with the endpoint URL of each daemon, so that the agent can access the URL and pull metrics on the health of the target daemon servers. Specify such URL in the Statestored URL text box . To know how to configure the eG agent with the API Endpoint URLs, refer to Configuring the eG agent with the API Endpoint URLs topic in Pre-requisites for Monitoring Apache Impala. |
Statestored Authorization type |
To access daemon server, you need to configure Digest authorization. Digest authorization is a method used to verify the identity of a user requesting access to API Endpoint URL of the daemon server. Digest authorization uses a combination of a username, password, and a unique message digest (a type of encrypted code). Specify such Digest authorization value in the Statestored Authorization type text box considering the below criteria:
By default, Statestored Authorization type value is no_auth. |
Statestored User, Statestored Password and Confirm Password |
Once you enable web server access to the eG agent, make sure that 'password authentication' is also enabled. This is needed, so that the eG agent is able to access the built-in web server for each daemon in a safe, secure manner. Configure the credentials of such a user against Statestored User and Password text boxes. Confirm the password by retyping it in the Confirm Password text box. To know how to configure the eG agent with 'secure' access to the built-in web server, refer to Configuring the eG agent with 'secure' access to the built-in web server topic in Pre-requisites for Monitoring Apache Impala. By default, Impalad user parameter is set to none.
|
Catalogd URL |
Each daemon server has a different endpoint URL. You need to configure the eG agent with the endpoint URL of each daemon, so that the agent can access the URL and pull metrics on the health of the target daemon servers. Specify such URL in the Catalogd URL text box . To know how to configure the eG agent with the API Endpoint URLs, refer to Configuring the eG agent with the API Endpoint URLs topic in Pre-requisites for Monitoring Apache Impala. |
Catalogd Authorization type |
To access daemon server, you need to configure Digest authorization. Digest authorization is a method used to verify the identity of a user requesting access to API Endpoint URL of the daemon server. Digest authorization uses a combination of a username, password, and a unique message digest (a type of encrypted code). Specify such Digest authorization value in the Catalogd Authorization type text box considering the below criteria:
By default, Statestored Authorization type value is no_auth. |
Catalogd User, Catalogd Password and Confirm Password |
Once you enable web server access to the eG agent, make sure that 'password authentication' is also enabled. This is needed, so that the eG agent is able to access the built-in web server for each daemon in a safe, secure manner. Configure the credentials of such a user against Catalogd User and Password text boxes. Confirm the password by retyping it in the Confirm Password text box. To know how to configure the eG agent with 'secure' access to the built-in web server, refer to Configuring the eG agent with 'secure' access to the built-in web server topic in Pre-requisites for Monitoring Apache Impala. By default, Impalad user parameter is set to none.
|
Measurement | Description | Measurement Unit | Interpretation |
---|---|---|---|
System memory reserved |
Indicates the reserved size of the system memory. |
MB |
|
Physical memory reserved |
Indicates the reserved size of the physical memory. |
MB |
The derived metric computing the amount of physical memory (in MB) used by the query process, includes the actual memory in use and free bytes reserved by TCMalloc but it does not include the TCMalloc metadata. |
Pageheap unmapped memory |
Indicates the size of the Pageheap unmapped memory. |
MB |
It represents the memory size of free and unmapped pages in page heap. These are bytes that have been released back to the OS, possibly by one of the MallocExtension "Release" calls. They can be used to fulfill allocation requests, but typically incur a page fault. They always count towards virtual memory usage, and depending on the OS, typically do not count towards physical memory usage. |
Memory in use |
Indicates the size of the used memory. |
MB |
It represents the memory size used by the application. This will not typically match the memory use reported by the OS, because it does not include TCMalloc overhead or memory fragmentation. Ideally, the value of this measure should be very low. |
Free pageheap memory |
Indicates the size of the Free pageheap memory. |
MB |
It represents the memory size of free and mapped pages in page heap. These bytes can be used to fulfill allocation requests. They always count towards virtual memory usage, and unless the underlying memory is swapped out by the OS, they also count towards physical memory usage. |