Instrumenting Docker Swarm for .NET Core Application Transaction Monitoring

Docker Swarm is a native container orchestration tool integrated into the Docker Engine that allows you to manage a cluster of Docker hosts as a single virtual system.

A "swarm" consists of multiple Docker hosts, known as nodes, which can be physical or virtual machines.

  • Manager Nodes: Act as the "control plane." They manage cluster state, handle orchestration, and schedule tasks.

  • Worker Nodes: Primarily receive and execute the tasks (containers) assigned to them by manager nodes.

  • Services & Tasks: A service is the definition of how containers should run (e.g., image name, number of replicas), while a task is an individual running container instance within that service.

The eG .NET Core BTM can be configured to profile and monitor the performance of .NET Core applications that are deployed as Docker Swarm services using a stack-based deployment.

The eG .NET Core Profiler integrates with these Swarm services using Host Bind Mounts.

Since Docker Swarm distributes containers across multiple nodes (Managers and Workers), profiler binaries must be pre-staged on the local storage of every node in the cluster.

Each application container mounts the profiler directory (/opt/egbtm) from the host using volume mapping. During startup, the .NET runtime loads the profiler library based on the environment variables defined in the stack YAML file. Every time a user transacts with the target .NET Core application, the .NET Core profiler traces the path of the transaction, measures the roundtrip time of that transaction, and also captures its responsiveness at each node in its path.

eG Enterprise auto-discovers the .NET Core application running in the Docker swarm as a Kestrel Web Linux component. Every time the .NET Core Business Transactions Test mapped to this component is executed, it communicates with the profiler via a configured TCP port, pulls the metrics of interest, and reports the same to the eG manager. Alerts are sent out if any transaction is unduly slow or is stalling, thus impacting overall user experience with the application. Cross-application transaction flows also help administrators to visualize the transaction path and accurately identify the node / pointcut where transaction performance degraded.

The broad steps for BTM-eabling a .NET Core application running in a Docker Swarm are as follows:

  • Installing the eG agent on each node in the Docker swarm;

  • Verifying the eG .NET Core Profiler's Setup on each node of the Docker swarm;

  • Modifying the stack YAML file of the target application

  • Deploying the target application's stack YAML

  • Verifying profiler attachment

Each of the above steps are discussed elaborately in the sections below.

Installing the eG Agent on Docker Swarm

To monitor a .NET core application that is running as a Docker swarm service, you have install the eG agent in each node of a Docker swarm. To achieve this, follow the steps below:

  1. Login to the Docker swarm node.

  2. Open a browser, connect to the eG manager, and login to the eG administrative interface.

  3. When Figure 1 appears, click the Discover/Monitor option in the left panel as indicated by Figure 1.

    Clicking Discover/Monitor Option

    Figure 1 : Selecting the Discover/Monitor option in the left panel

  4. Figure 2 will then appear. Click on Kubernetes and Containers in the right panel of Figure 2.

    Selecting Kubernetes and Containers

    Figure 2 : Clicking on the Kubernetes and Containers option

  5. This will open Figure 3. Click on Standalone Docker, as indicated by Figure 3.

    Clicking on Standalone Docker

    Figure 3 : Clicking on the Standalone Docker option

  6. Figure 4 will then appear. Click on the Copy button indicated by Figure 4 to copy the command above.

    Figure 4 : Copying the command to run on the Docker swarm node

  7. Open the Shell prompt on the Docker Swarm node and paste the copied command there to run it.

  8. Once the command runs successfully, the eG agent will be installed on the Docker swarm node.

  9. Repeat steps 1-8 on each Docker swarm node to install the eG agent on it.

  10. Next, verify that the container agent has been installed successfully. For this purpose, open the Shell prompt on the Docker swarm node and issue the following command:

    docker ps

  11. Check if this command outputs a container with IMAGE name, eginnovations/agent:7.5.4-fpl

  12. If it does, you can conclude that the container agent has been successfully installed on that node.

  13. Repeat steps 10-12 to verify successful agent installation on each node.

  14. Next, proceed to check if the eG agent has been able to successfully discover and auto-manage the Docker swarm node as a Docker Host and the .NET Core application running on that node as a Kestrel Web Linux component. For that, log into the eG management console once again.

  15. Look for Docker Host and Kestrel Web Linux components in the Components At-A-Glance section (see Figure 5 and Figure 6). If these component types suddenly appear in the Components At-A-Glance window after the container agent was installed on a Docker swarm node, it is a clear indicator that the agent has successfully discovered and managed the said components.

    Confirming Docker Host Discovery

    Figure 5 : Verifying that the Docker swarm node has been auto-discovered as a Docker Host

    Confirming Kestrel Web Linux Discovery

    Figure 6 : Verifying that the .NET Core application running in the Docker swarm has been managed as a Kestrel Web Linux component

  16. Net, proceed to configure the .NET Core Business Transactions Test for the managed Kestrel Web Linux component, so that it is ready to collect performance metrics on transaction health by communicating with the eG .NET Core Profiler. To know how to configure this test, refer to .NET Core Business Transactions Test.

Verifying the eG .NET Core Profiler's Setup on the Docker Swarm Nodes

As soon as the eG agent is installed on a Docker swarm node, the eG .NET Core profiler will also be automatically installed and configured on that node. To verify that the profiler has been successfully setup, do the following:

  1. Login to the target node in the Docker swarm.

  2. The eG agent installation will automatically create a directory named /opt/egbtm will on the node, This directory is used by the eG .NET Core profiler for storing profiler binaries and configuration files.

  3. From the Shell prompt, issue the following commands one after another:

    cd /opt/egbtm/dotnet/config

    ls

  4. This will output the list of files in the /opt/egbtm/dotnet/config folder. Typically, the profiler's configuration files will be stored in this folder. Check if the commands above list the following ini files:

    eG_CoreclrBTM.ini

    eG_CoreclrServer.ini

    If they do, then you can confirm the profiler has been configured correctly.

  5. Next, issue the following commands one after another at the prompt:

    cd /opt/egbtm/dotnet/centos

    ls

  6. This will list the files in the /opt/egbtm/dotnet/centos folder. The profiler's *.so file and the profiler library will be stored in this folder. Check if the commands above list the following files:

    eGCoreClrHelperAssembly.dll

    eGCoreCLRMonitor.so

    If they do, then you can confirm that the profiler has been setup properly.

Modifying the Target Application's Stack YAML File

Next, you should edit the application's stack YAML file to include profiler-related configurations. For that, do the following:

  1. Open the stack YAML file of the target application in an Editor. Typically, when creating a stack YAML, a user can give any name they choose to that YAML and any name to the stack. For instance, if the stack is named dotnet-btm, then its corresponding YAML can be named, dotnet-btm-stack.yml.

  2. First, include a volumes section in the stack YAML file, as indicated below:

    volumes:

    - /opt/egbtm:/opt/egbtm

    This entry maps the /opt/egbtm folder from the host machines to the container so the application can access the profiler library.

  3. Next include an environment: section in the YAML file, as shown below:

    environment:

  4. Here, first enable transaction profiling of the target .NET Core application using the variable CORECLR_ENABLE_PROFILING.

    CORECLR_ENABLE_PROFILING: "1"

  5. Next, configure the CORECLR_PROFILER variable with the GUID of the profiler, as shown below:

    CORECLR_PROFILER: "{950A85A8-9E1B-4611-BEA6-E5A20686387E}"

  6. Then, specify the CORECLR_PROFILER_PATH variable. This is where you configure the full path to the .NET Core profiler software.

    CORECLR_PROFILER_PATH: "/opt/egbtm/dotnet/centos/eGCoreCLRMonitor.so"

  7. Proceed to indicate the level of granularity of the debug messages that should be captured by the profiler. Specify the value 1 for messages that offer less details, and specify 3 for detailed messages.

    EG_PRFDEBUG_LEVEL: "3"

  8. The eG agent attempts to establish a connection with the profiler via each of the ports configured against EG_COLLECTOR_DOTNET_PORT, until it succeeds. To facilitate this, provide a comma-separated list of ports against EG_COLLECTOR_DOTNET_PORT.

    EG_COLLECTOR_DOTNET_PORT: "<Comma-separated list of ports>"

    For instance, your specification can be:

    EG_COLLECTOR_DOTNET_PORT: "11003, 12003, 13003, 14003, 15003, 16003, 17003, 18003, 19003"

  9. To enable the container application to communicate with the host machine where the eG agent is running, set EG_COLLECTOR_IP as host.docker.internal.

    EG_COLLECTOR_IP: host.docker.internal

  10. Next, to ensure that the EG_COLLECTOR_IP, host.docker.internal, actually works inside the container, include the extra_hosts parameter. Set this parameter to host-gateway; this is a special Docker value that maps to the host machine’s gateway IP. This configuration is needed because, on Linux (especially in Docker Swarm), host.docker.internal is not available by default. Since the extra_hosts parameter adds an entry to the container’s /etc/hosts file for host-gateway, it makes sure your container can resolve host.docker.internal correctly.

    extra_hosts:

    - "host.docker.internal:host-gateway"

  11. Finally, save the file.

Deploying the Target Application's Stack YAML File

Execute the following steps on the docker swarm node to deploy the application and verify that the service is running correctly:

docker stack deploy -c <compose_file>.yml <stack_name>

Here:

<compose-file>.yml represents the name of the stack configuration file

<stack_name> is the user-defined name of the stack

A sample command for stack dotnet-btm is given below:

docker stack deploy -c dotnet-btm-stack.yml dotnet-btm

Confirm Profiler Attachment

To verify that the profiler is successfully attached and actively monitoring the application, inspect the logs of a running application container.

Before that, obtain the container ID by issuing the following command at the Shell prompt:

docker ps

Once you get the container ID, proceed to issue the following command to view the logs:

docker logs -f <container_ID>

For instance, if container ID is 0607ff79b46d, then the command will be:

docker logs -f 0607ff79b46d

This will open the docker logs. Search the logs for messages that start with the string, [eGDotNetBTM]. If you find such messages in the logs, you can conclude that the profiler has been successfully loaded into the application container.