K8s Namespaces Test

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.

Namespaces are intended for use in environments with many users spread across multiple teams, or projects. Namespaces are a way to divide cluster resources between multiple users (via resource quota). A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the API resources - i.e., the quantity of objects (eg., pods, services, deployments etc.) that can be created in a namespace , as well as the total amount of compute resources - i.e., CPU and memory - that may be consumed by the API resources in that namespace.

If quota is enabled in a namespace for compute resources like CPU and memory, users must specify requests and/or limits for those values. A request is the amount of that resource that the system will guarantee to the namespace, and a limit is the maximum amount that the system will allow the namespace to use. Typically, within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace’s resource quota. You can also define compute resource usage limits for individual containers, during their creation. However, to ensure that no single pod/container in a namespace hogs the resources of that namespace, you can define a Limit Range. Limit Range is a policy to constrain resource by Pod or Container in a namespace. A limit range provides constraints that can:

Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.
Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.
Enforce a ratio between request and limit for a resource in a namespace.
Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime. The default requests/limits will apply to those containers for which requests and/or limits have not been specifically defined.

If creating or updating an API resource in a namespace violates a quota constraint / limit range, then that create/update request will fail. For instance, say a namespace is configured with a resource quota that restricts the number of Pods that can be created in that namespace to 2. In this case, if the creation of a third Pod is attempted within that namespace, then the pod creation will fail. Likewise, if you are attempting to create a container with a memory limit of 2Gi within a namespace that has a resource quota constraint of 1Gi, then Kubernetes will not allow the container to be created. This can eventually result in a mismatch between the cluster's desired state and its actual state. To avoid this, administrators must first be well-aware of the resource request/limit that has been set per namespace and also for the pods and containers in each namespace. Then, administrators should track how the containers in each namespace are using the allocated compute resources, and determine whether any namespace is likely to violate its quota, well before an actual violation happens. The Kube Namespaces enables administrators to pre-empt any adverse impact to cluster health by monitoring namespaces, their quota definitions, and their resource usage!

This test auto-discovers the namespaces configured in a Kubernetes/OpenShift cluster, and reports the current state of each namespace, thus bringing inactive/terminating namespaces to light. Additionally, the test also reports the request/limit settings for each namespace and the requests/limits that apply to the pods and the containers in every namespace. Furthermore, the test also measures how much of the allowed / guaranteed compute resources each namespace is currently utilizing, thus enabling administrators to accurately identify the namespaces that are currently experiencing or may potentially experience a contention for resources. The resource quota of such namespaces may require rework. This way, the test proactively alerts administrators to problem conditions that may be caused by poor resource quota definitions, and prompts them to initiate preventive action.

Target of the test : A Kubernetes/OpenShift Cluster

Agent deploying the test : A remote agent

Outputs of the test : One set of results for each namespace configured in the Kubernetes/OpenShift cluster being monitored

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	The IP address of the host for which this test is to be configured.
Port	Specify the port at which the specified Host listens. By default, this is 6443.
Load Balancer / Master Node IP	To run this test and report metrics, the eG agent needs to connect to the Kubernetes API on the master node and run API commands. To enable this connection, the eG agent has to be configured with either of the following: If only a single master node exists in the cluster, then configure the eG agent with the IP address of the master node. If the target cluster consists of more than one master node, then you need to configure the eG agent with the IP address of the load balancer that is managing the cluster. In this case, the load balancer will route the eG agent's connection request to any available master node in the cluster, thus enabling the agent to connect with the API server on that node, run API commands on it, and pull metrics. By default, this parameter will display the Load Balancer / Master Node IP that you configured when manually adding the Kubernetes/OpenShift cluster for monitoring, using the Kubernetes Cluster Preferences page in the eG admin interface (see Figure 3). The steps for managing the cluster using the eG admin interface are discussed elaborately in How to Monitor the Kubernetes/OpenShift Cluster Using eG Enterprise? Whenever the eG agent runs this test, it uses the IP address that is displayed (by default) against this parameter to connect to the Kubernetes API. If there is any change in this IP address at a later point in time, then make sure that you update this parameter with it, by overriding its default setting.
SSL	By default, the Kubernetes/OpenShift cluster is SSL-enabled. This is why, the eG agent, by default, connects to the Kubernetes API via an HTTPS connection. Accordingly, this flag is set to Yes by default. If the cluster is not SSL-enabled in your environment, then set this flag to No.
Authentication Token	The eG agent requires an authentication bearer token to access the Kubernetes API, run API commands on the cluster, and pull metrics of interest. The steps for generating this token have been detailed in How Does eG Enterprise Monitor a Kubernetes/OpenShift Cluster? Typically, once you generate the token, you can associate that token with the target Kubernetes/OpenShift cluster, when manually adding that cluster for monitoring using the eG admin interface. The steps for managing the cluster using the eG admin interface are discussed elaborately in How to Monitor the Kubernetes/OpenShift Cluster Using eG Enterprise? By default, this parameter will display the Authentication Token that you provided in the Kubernetes Cluster Preferences page of the eG admin interface, when manually adding the cluster for monitoring (see Figure 3). Whenever the eG agent runs this test, it uses the token that is displayed (by default) against this parameter for accessing the API and pulling metrics. If for any reason, you generate a new authentication token for the target cluster at a later point in time, then make sure you update this parameter with the change. For that, copy the new token and paste it against this parameter.
Proxy Host	If the eG agent connects to the Kubernetes API on the master node via a proxy server, then provide the IP address of the proxy server here. If no proxy is used, then the default setting -none - of this parameter, need not be changed,
Proxy Port	If the eG agent connects to the Kubernetes API on the master node via a proxy server, then provide the port number at which that proxy server listens here. If no proxy is used, then the default setting -none - of this parameter, need not be changed,
Proxy Username, Proxy Password, Confirm Password	These parameters are applicable only if the eG agent uses a proxy server to connect to the Kubernetes/OpenShift cluster, and that proxy server requires authentication. In this case, provide a valid user name and password against the Proxy Username and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box. If no proxy server is used, or if the proxy server used does not require authentication, then the default setting - none - of these parameters, need not be changed.
DD Frequency	Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 3:1. This indicates that, by default, detailed measures will be generated every third time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency.
Detailed Diagnosis	To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled: The eG manager license should allow the detailed diagnosis capability Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measurements made by the test

Measurement

Description

Measurement Unit

Interpretation

Status

Indicates the current status of this namespace

The values that this measure reports and their corresponding numeric values are detailed in the table below:

Measure Value	Numeric Value
Active	1
Terminating	0

Note:

By default, this test reports the Measure Values listed in the table above to indicate the state of a namespace. In the graph of this measure however, the state is indicated using the numeric equivalents only.

Use the detailed diagnosis of this measure to view the labels that have been configured for the objects in an namespace. Labels are key/value pairs that are attached to objects, such as pods. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system. Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time. Each object can have a set of key/value labels defined. Example of labels: "release" : "stable", "release" : "canary"

Time since namespace creation

Indicates how old the namespace is

The value of this measure is expressed in number of days, hours, and minutes.

Total pods in namespace

Indicates the number of pods in this namespace.

Number

A Pod is the basic execution unit of a Kubernetes application–the smallest and simplest unit in the Kubernetes object model that you create or deploy. A Pod encapsulates an application’s container (or, in some cases, multiple containers), storage resources, a unique network IP, and options that govern how the container(s) should run. A Pod represents a unit of deployment: a single instance of an application in Kubernetes, which might consist of either a single container or a small number of containers that are tightly coupled and that share resources.

To know which are the pods in a namespace, use the detailed diagnosis of this measure.

If the resource quota enforced on a namespace restricts the number of pods you can create in a namespace, then you can use this measure to ascertain how much of that quota is being used currently, and how many more pods you can create before the quota is fully exhausted. If the node on which the existing pods are running has the resource capacity to support more pods, you may want to change your quota accordingly.

Total services in namespace

Indicates the number of services in this namespace.

Number

In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service).

To know which services are in a namespace, use the detailed diagnosis of this measure.

If the resource quota enforced on a namespace restricts the number of services you can create in a namespace, then you can use this measure to ascertain how much of that quota is being used currently, and how many more services you can create before the quota is fully exhausted.

Maximum CPU limits in container

Indicates the CPU resource limit set in the Limit Range for containers in this namespace.

Millicpu

These measures will be reported only if a Limit Range has been configured and enabled for the containers in a namespace, and CPU limits/requests have been configured in that Limit Range.

Typically, to limit consumption by individual containers in a namespace, a Limit Range specification is used.

If a Limit Range is enforced on the containers in a namespace, then the resource consuming capacity of each container in that namespace will be determined by the min/max limits defined within that Limit Range. In this case therefore, the max and min limit settings in the Limit Range will be reported as values of these measures, respectively.

Moreover, in this case, Kubernetes will automatically foil any attempt to create/update a container, if that operation ends up violating the limit set in the Limit Range. For instance, say that the Limit Range specification enforced on a namespace rules that no single container in that namespace should consume CPU over 800m (max limit) or lesser than 100m (min limit). In this case, if an attempt is made to create a single container with a CPU limit of 900m, then Kubernetes will automatically foil that attempt. This is because, that container violates the max limit of 800m that is set per container in the enforced Limit Range. The container creation fails, even if that container does not violate the total CPU consumption limit set in the resource quota of that namespace.

Minimum CPU requests in container

Indicates the minimum CPU request limit set in the Limit Range for containers in this namespace.

Millicpu

Default CPU limits in container

Indicates the default CPU limit defined in the Limit Range for containers in this namespace.

Millicpu

These measures will be reported only if a if default CPU requests/limits are set in the Limit Range for the containers in a namespace.

Default CPU requests and limits can be set for the containers in a pod, using the Limit Range specification. These default settings apply only when minimum and/or maximum CPU limits are not defined at the individual container-level (during container creation).

For instance, if a container being created is not configured with a CPU limit, but is configured with a CPU request, then the default CPU limit configured in the Limit Range will apply to that container. If a container being created is not configured with a CPU request, but is configured with a CPU limit instead, then Kubernetes does not automatically apply the default CPU request configured in the Limit Range to that container. Instead, the limits set for that container during creation are automatically set as its requests. On the other hand, if a container is being created with neither CPU limits nor CPU requests defined, then the default CPU limit and request defined in the Limit Range will automatically apply.

Default CPU requests in container

Indicates the default CPU request setting defined in the Limit Range for containers in this namespace.

Millicpu

Maximum memory limits in container

Indicates the memory resource limit set in the Limit Range for containers in this namespace,

These measures will be reported only if a Limit Range has been configured and enabled for the containers in a namespace, and memory limits/requests have been configured in that Limit Range.

Typically, to limit consumption by individual containers in a namespace, a Limit Range specification is used.

If a Limit Range is enforced on a namespace, then the resource consuming capacity of each container in that namespace will be determined by the min/max limits defined within that Limit Range. In this case therefore, the max and min memory limit settings in the Limit Range will be reported as values of these measures, respectively. Moreover, in this case, Kubernetes will automatically foil any attempt to create/update a container that violates the limit set in the Limit Range. For instance, say that the Limit Range specification enforced on a namespace rules that no single container in that namespace should consume memory over 500 MB (max limit) or lesser than 100 MB (min limit). In this case, if an attempt is made to create a single container with a memory limit of 800 MB, then Kubernetes will automatically foil that attempt. This is because, that container violates the max limit of 500 MB that is set per container in the enforced Limit Range. The container creation fails, even if that operation does not violate the total memory consumption limit set in the resource quota of that namespace.

If the value of the Maximum memory limits in container measure reports the value 0, it means that the containers in the namespace have no upper bound on the amount of memory they use. In such situations, the Container could use all of the memory available on the Node where it is running which in turn could invoke the OOM Killer. Further, in case of an OOM Kill, a container with no resource limits will have a greater chance of being killed.

Minimum memory requests in container

Indicates the minimum memory request limit set in the Limit Range for containers in this namespace.

Default memory limits in container

Indicates the default memory limit defined in the Limit Range for containers in this namespace.

These measures will be reported only if a if default memory requests/limits are set in the Limit Range enforced for the containers in a namespace.

Default memory requests and limits can be set for the containers in a pod, using the Limit Range specification. These default settings apply only when minimum and/or maximum memory limits are not defined at the individual container-level (during container creation).

For instance, if a container being created is not configured with a memory limit, but is configured with a memory request, then the default memory limit configured in the Limit Range will apply to that container. If a container being created is not configured with a memory request, but is configured with a memory limit instead, then Kubernetes does not automatically apply the default memory request configured in the Limit Range to that container. Instead, the limits set for that container during creation are automatically set as its requests. On the other hand, if a container is being created with neither memory limits nor memory requests defined, then the default memory limit and request defined in the Limit Range will automatically apply.

Default memory requests in container

Indicates the default memory request setting defined in the Limit Range for containers in this namespace.

CPU limits

Indicates the total amount of CPU resources that containers in this namespace are allowed to use, as per the resource quota.

Millicpu

Resource requests/limits set using the ResourceQuota object govern the aggregate resource consumption of a namespace - i.e., the total resources that can be consumed/requested across all pods/containers in a namespace.

A resource quota is violated only when the total consumption of a resource, across pods/containers in the namespace, exceeds the limits defined in the resource quota.

For instance, say that the resource quota of a namespace enforces a CPU usage limit of 2 cores and a memory usage limit of 500Gi. In this case, Kubernetes will allow you to create 2 containers with a CPU core each and 100Gi of memory each. However, if an attempt is made to create another container configured with 1 CPU core and 200Gi of memory, then such an addition operation will fail. This is because, the addition increases the total CPU usage of the namespace to 3 CPU cores, which violates the 2 core limit set by the resource quota.

CPU requests

Indicates the minimum amount of CPU resources that is guaranteed to the containers in this namespace, as per the resource quota.

Millicpu

Memory limits

Indicates the total amount of memory resources that containers in this namespace are allowed to use, as per the resource quota.

Memory requests

Indicates the minimum amount of memory resources that is guaranteed to the containers in this namespace, as per the resource quota.

Used CPU limits

Indicates the sum of the CPU limits configured for the containers in this namespace.

Millicpu

If a resource quota is enabled for a namespace , you may want to compare the value of this measure with that of the CPU limits for that namespace. If this comparison reveals that the value of this measure is equal to or close to that of the CPU limits measure, it implies that that namespace has or is about to exhaust its quota of CPU resources. If the node on which the containers are running is resource-thick, you may want to reconfigure the resource quota and increase the aggregate CPU consumption capacity of the namespace, so as to prevent a resource quota violation and consequent throttling of creation/updation operations on the namespace.

Used memory limits

Indicates the sum of the memory limits configured for the containers in this namespace.

If a resource quota is enabled for a namespace, you may want to compare the value of this measure with that of the Memory limits measure for that namespace. If this comparison reveals that the value of this measure is equal to or close to that of the Memory limits measure, it implies that that namespace has or is about to exhaust its quota of memory resources. If the node on which the containers are running is resource-thick, you may want to reconfigure the resource quota and increase the aggregate memory consumption capacity of the namespace, so as to prevent a resource quota violation and consequent throttling of creation/updation operations on the namespace.

Used CPU requests

Indicates the sum of the CPU requests configured for the containers in this namespace.

Millicpu

If a resource quota is enabled for a namespace, you may want to compare the value of this measure with that of the CPU requests measure for that namespace. If this comparison reveals that the value of this measure is equal to or close to that of the CPU requests measure, it implies that that namespace is rapidly utilizing the CPU resources guaranteed to it. If the node on which the containers are running is resource-thick, you may want to reconfigure the resource quota and increase the aggregate requests for the namespace, so as to prevent a resource quota violation and consequent throttling of creation/updation operations on the namespace.

Used memory requests

Indicates the sum of the memory requests configured for the containers in this namespace.

If a resource quota is enabled for a namespace , you may want to compare the value of this measure with that of th Memory requests measure for that namespace. If this comparison reveals that the value of this measure is equal to or close to that of the Memory requests measure, it implies that that namespace is rapidly utilizing the memory resources guaranteed to it. If the node on which the containers are running is resource-thick, you may want to reconfigure the resource quota and increase the aggregate memory requests for the namespace, so as to prevent a resource quota violation and consequent throttling of creation/updation operations on the namespace.

Maximum CPU limits in pod

Indicates the maximum CPU usage limit set in the Limit Range for pods in this namespace.

Millicpu

These measures will be reported only if a Limit Range has been configured and enabled for the pods in a namespace, and CPU limits/requests have been configured in that Limit Range.

Typically, to limit consumption by individual pods in a namespace, a Limit Range specification is used.

If a Limit Range is enforced on a namespace, then the resource consuming capacity of each pod in that namespace will be determined by the min/max limits defined within that Limit Range. In this case therefore, the max and min limit settings in the Limit Range will be reported as values of these measures, respectively. Moreover, in this case, Kubernetes will automatically foil any attempt to create/update a container, if that operation ends up violating the limit set in the Limit Range for pods in the namespace. For instance, say that the Limit Range specification enforced on a namespace rules that no single pod in that namespace should consume CPU over 800m (max limit) or lesser than 100m (min limit). In this case, if an attempt is made to create a single pod containing containers with a total CPU limit of 900m, then Kubernetes will automatically foil that attempt. This is because, that pods violates the max limit of 800m that is set per pod in the enforced Limit Range. The pod creation fails, even if that pod does not violate the total CPU consumption limit set in the resource quota of that namespace.

Minimum CPU requests in pod

Indicates the minimum amount of CPU resources guaranteed to the pods in this namespace.

Millicpu

Maximum memory limits in pod

Indicates the maximum memory usage limit set in the Limit Range for pods in this namespace.

These measures will be reported only if a Limit Range has been configured and enabled for the pods in a namespace, and memory limits/requests have been configured in that Limit Range.

To limit consumption by individual pods in a namespace, a Limit Range specification is used.

If a Limit Range is enforced on a namespace, then the resource consuming capacity of each pod in that namespace will be determined by the min/max limits defined within that Limit Range. In this case therefore, the max and min memory limit settings in the Limit Range will be reported as values of these measures, respectively. Moreover, in this case, Kubernetes will automatically foil any attempt to create/update a pod, if that operation may potentially violate the limit set in the Limit Range. For instance, say that the Limit Range specification enforced on a namespace rules that no single pod in that namespace should consume memory over 500 MB (max limit) or lesser than 100 MB (min limit). In this case, if an attempt is made to create a single pod containing containers with a total memory limit of 800 MB, then Kubernetes will automatically foil that attempt. This is because, that pod violates the max limit of 500 MB that is set per pod in the enforced Limit Range. The pod creation fails, even if that operation does not violate the total memory consumption limit set in the resource quota of that namespace.

If the value of the Maximum memory limits in pod measure reports the value 0, it means that the pods in the namespace have no upper bound on the amount of memory they use. In such situations, the pods could use all of the memory available on the Node where it is running which in turn could invoke the OOM Killer. Further, in case of an OOM Kill, a pod with no resource limits will have a greater chance of being killed.

Minimum memory requests in container

Indicates the minimum amount of memory resources guaranteed to the containers in this namespace.

The detailed diagnosis of the Total pods measure reveals the names of the pods in the namespace, the IP address of the pods, and the node on which each pod is running.

Figure 1 : The detailed diagnosis of the Total Pods measure

The detailed diagnosis of the Total services measure reveals the names of the services in the namespace.

Figure 2 : The detailed diagnosis of the Total services measure