{"id":36175,"date":"2025-01-23T09:41:25","date_gmt":"2025-01-23T14:41:25","guid":{"rendered":"https:\/\/www.eginnovations.com\/blog\/?p=36175"},"modified":"2025-03-18T06:43:19","modified_gmt":"2025-03-18T10:43:19","slug":"designing-for-scale-how-eg-enterprise-manages-millions-of-metrics-with-aiops-driven-self-monitoring","status":"publish","type":"post","link":"https:\/\/www.eginnovations.com\/blog\/designing-for-scale-how-eg-enterprise-manages-millions-of-metrics-with-aiops-driven-self-monitoring\/","title":{"rendered":"Designing for Scale: How eG Enterprise Manages Millions of Metrics with AIOps-driven Self-Monitoring"},"content":{"rendered":"
\n

Customers evaluate a modern observability and monitoring solution by the ROI they get, self-monitoring capabilities ultimately improve scalability and quality. The value of any observability solution lies in its ability to proactively detect and alert customers to issues before they cause a business-impacting outage. IT infrastructures and applications can fail in many different ways. A run-away process, a memory leak in an application, an application that fails to release database connections, a mis-configured system, etc. can all result in IT outages, and therefore, it is imperative that a modern observability solution<\/a> collects and analyzes a wide range of different metrics. For a single system, thousands of metrics may have to be collected and analyzed in real-time. For a large infrastructure with thousands of systems and applications, it is no surprise that eG Enterprise has to collect and analyze millions of metrics.<\/p>\n

\"Screenshot<\/p>\n

Figure 1: eG Enterprise collecting over 12million metrics in a real-world infrastructure<\/div>\n

Figure 1 shows a snapshot of the eG Enterprise console of a production deployment managing over 6000 components. 12 million measurements is a lot of metrics, logs, traces and tests! Far more than a human operator can process or overview and only possible because of powerful AIOps-engine<\/a> at the heart of eG Enterprise. Without AIOps capabilities, the burden of configuring thresholds for each metric, manually setting alerts and tracking them over time is put on the human administrator \u2013 an impossible job!<\/p>\n

Scaling an observability platform to process tens of millions of events, metrics, logs, and traces daily is no small feat. eG Enterprise achieves this with a combination of efficient design principles, AIOps-powered automation, and self-monitoring capabilities that ensure continuous optimization.<\/p>\n

We also have to pay considerable attention as to how this data gets processed and presented into human-consumable overviews. This blog covers how eG Enterprise scales to collect, analyze and report on millions of metrics.<\/p>\n

<\/span>Step 1: Design to Minimize Inefficiencies<\/span><\/h2>\n

A distributed monitoring approach is used to scale the monitoring. As the target infrastructure scales, you can add additional \u201cremote\u201d monitors for agentless monitoring of the infrastructure. Multi-threading concepts are used to ensure that the monitoring can scale as required. And you can configure the remote monitors to operate in a highly available cluster to avoid a single point of failure.<\/p>\n

While the management server is the heart of the eG Enterprise system, storing and analyzing metrics, the actual analysis of the metrics is performed by the agents. This distributes the workload across the infrastructure, minimizing the chance of the eG Enterprise manager becoming a single point bottleneck.<\/p>\n

eG Enterprise also uses a concept of management by exception. Its state analysis and alarm correlation engine<\/a> only analyzes problematic metrics and correlates them, thereby minimizing the analysis effort that it performs.<\/p>\n

The data processing, analysis and user interfacing components are segregated from one another, ensuring that there\u2019s minimum interference between these components. Modern data processing mechanisms such as data partitioning<\/a> are used to ensure that even if the database grows to many Terra-bytes, the monitoring system remains functioning at its peak.<\/p>\n

To minimize data collection and storage overheads, the default models of the system are designed with scalability in mind. For example, there is no point sampling Azure billing metrics every 3 seconds as they are only updated a few times a day<\/a>.<\/p>\n

While scalability of the management system and the agents are important, it is equally important for the communication between the management system and agents to be efficient. Inefficiencies can result in overloading the network, impacting the performance of the very infrastructure that the monitoring system is supposed to oversee. eG Enterprise leverages caching where appropriate to ensure that configurations are not repeatedly communication. Compression techniques are used to minimize bandwidth usage for metrics storage, accelerating performance and lowering costs.<\/p>\n

eG Enterprise continually self-monitors every deployment so we get continual feedback and a deep understanding of how the platform behaves under different customer use cases. These insights and understandings from self-monitoring are fed into iterations of the solution\u2019s architecture.<\/p>\n

<\/span>Step 2: Scale Horizontally as well as Vertically<\/span><\/h2>\n

\"imageFor a scalable architecture, it is recommended to have the management server and database on separate systems. In a cloud deployment, a database service like Amazon RDS or Azure SQL can be used as well. Compute and data processing resources in the eG Enterprise architecture can be distributed and scaled as required, by allocating additional compute, memory and storage resources. A sizing calculator is available to compute the resources needed for the management server and database.<\/p>\n

The \u201cremote\u201d monitors used for agentless monitoring can also be scaled up if required to monitor hundreds of devices or virtual machines. Multi-threading is used extensively on the manager and agents to achieve scale up.<\/p>\n

\"imageHorizontal scaling is supported by enabling multiple eG Managers to operate in parallel and report to a central Super Manager<\/a>. This hierarchical setup allows organizations to monitor expansive IT environments efficiently by distributing workloads across several eG Managers, each handling specific segments of the infrastructure. The Super Manager aggregates data from all eG Managers, providing a unified view of performance and health metrics without overloading individual components.<\/p>\n

Horizontal scaling (also described as scaling out) is fundamental to how cloud platforms scale \u2013 we’ve some history and background on the fundamentals of scaling methodologies available here: Understanding Scale Up vs. Scale Out – And Why You Need to Understand Scale Up vs. Scale Out to Be a Nutanix or HCI Guru | eG Innovations Understanding scale up vs. scale out \u2013 and why do you need to understand scale up vs. scale out to be a Nutanix or HCI Guru?<\/a>.<\/div>\n

Note that the data collection, analysis and retention is still performed by the individual managers, to ensure that the SuperManager does not become a single point bottleneck. The SuperManager configuration can be used for scalability \u2013 to handle tens of thousands of systems and hundreds of thousands of devices and VMs. This architecture can also be used for efficiency when handling geographically distributed environments. The agents can report to the manager local to them, ensuring that data does not leave their local region and minimizing bandwidth usage. At the same time, this architecture also allows for distributed administration: each region administers and manages their local deployment and the SuperManager is mainly for consolidated monitoring across regions.<\/p>\n

<\/span>Step 3: Code-level Efficiency \u2013 Adhere to Best Practice and then Self-Monitor<\/span><\/h2>\n

eG Enterprise makes extensive use of Java technologies, and we adhere to Java best practices for scalability and efficiency. Best practices we follow are well documented in our earlier blogs and include:<\/p>\n