{"id":26649,"date":"2022-09-13T07:24:44","date_gmt":"2022-09-13T11:24:44","guid":{"rendered":"https:\/\/www.eginnovations.com\/blog\/?p=26649"},"modified":"2025-01-31T06:37:00","modified_gmt":"2025-01-31T11:37:00","slug":"is-azure-down-proactive-alerting-for-azure-outages","status":"publish","type":"post","link":"https:\/\/www.eginnovations.com\/blog\/is-azure-down-proactive-alerting-for-azure-outages\/","title":{"rendered":"Is Azure Down? – Proactive Alerting for Azure Outages"},"content":{"rendered":"
\n

Last week (7th September 2022) I spotted a post from EUC and Cloud Veteran Marius Sandbu alerting his professional and social media networks to an outage of Azure Front Door that in turn meant a significant number of Azure services and resources were unavailable.<\/p>\n

\n

<\/span>Marius Posted:<\/span><\/h2>\n

\u201cHow much do you trust the cloud providers?<\/em><\/p>\n

About 1 hour ago, Microsoft started to have issues with their Front door service which affects many of their global services including Azure Portal, Azure Virtual Desktop, Azure AD, GitHub, and of course services that are placed behind Azure Front door. <\/em><\/p>\n

While outages like this can happen, ensure that you have some proper monitoring that monitors the cloud providers and services from an external point of view.\u201d <\/em><\/p>\n<\/div>\n

And of course, as ever, he is correct. It is best practice to monitor the availability of cloud services for yourself. Whilst Microsoft do maintain an Azure Status page<\/a> that notifies users of known outages, it can take a very long time for outages to actually show up on this site. Microsoft spend a lot of time verifying, reproducing and confirming issues, their regional extent, what other services impacted etc. As a result, an organization with end-users using AVD (Azure Virtual Desktop) can find their own help desk bombarded by support calls, emails, and tickets, long before Microsoft publish and confirm an outage. Slowdowns and more subtle performance issues can take even longer for any official information to appear.<\/p>\n

Organizations that proactively verify Azure availability and performance themselves have an advantage. If an Azure service is found to be down, IT can proactively broadcast that information to their users and warn help desk staff \u2013 a simple \u201cSorry, Microsoft Azure is down, we\u2019ve notified Microsoft and have to wait for them to fix it\u201d can pre-empt help desk tickets.<\/p>\n

Receiving early notification of outages and their extent also allows IT to direct users to alternatives that may be available, perhaps on-premises, in a different cloud or a different cloud region.<\/p>\n

<\/span>Azure Outage: What happened on the 7th September 2022?<\/span><\/h2>\n

Microsoft have already published their initial analysis: Preliminary Post Incident Review (PIR) – Azure Front Door – Connectivity Issues (Tracking ID YV8C-DT0),<\/a> that states:<\/p>\n

Between 16:10 and 19:55 UTC on 07 Sep 2022, subset of customers using Azure Front Door might have experienced connectivity issues. This could also be impacting customers\u2019 ability to access other Azure services that leverage Azure Front Door, this includes the Azure Management Portal and Azure CDN.<\/em><\/p>\n

The report includes some details of what went wrong and how they attempted to resolve the issue and bring services back online.<\/p>\n

<\/span>How eG Enterprise Proactively Monitors Azure Outages<\/span><\/h2>\n

At eG Enterprise there is no doubt as to whether we \u201cmight\u201d or \u201ccould\u201d have been affected by this issue because our proactive monitoring of our own Azure deployments detected the issue within minutes and allowed us to assess the impact.<\/p>\n

\"Azure<\/a><\/p>\n

Figure 1: The eG Enterprise alarm console – at 16:20 our synthetic AVD logon simulator detected issues, over the next few minutes web logins for key services such as Azure Active Directory and the Azure Subscription became unavailable. Note next to AVD Logon Simulator alert, a diagnosis icon appeared (magnifying glass).<\/div>\n

Exploring the AVD Logon Simulator alert, raised at 16:20, the detailed diagnosis confirmed the issue. A simple intuitive GUI suitable for L1\/L2 help desk staff showed the problem clearly lay with Web URL Access to Azure and not the authentication process.<\/p>\n

\"Detecting<\/a><\/p>\n

Figure 2: Proactive detailed diagnosis of the simulator logon failure<\/div>\n

Over the duration of the incident, eG Enterprise continued to probe Azure availability.<\/p>\n

\"azure-web-availability\"<\/a><\/p>\n

Figure 3: Azure Web Access\/Web Availability during the outage. Note: the GUI offers configurable options to zoom into the time period of interest.<\/div>\n

As you can see from the above graph of availability it was not a continual outage and services went up and down as Microsoft attempted to stabilize the system.<\/p>\n

This type of data is extremely useful when debating outage credit with cloud providers, especially smaller clouds, Microsoft are generally very good at crediting customers after outages. This data can also be extremely useful if working with support at a cloud vendor as it allows you to explain the impact of changes, they may make to help you correlating changes such as moving server to end-user availability etc.<\/p>\n

It is also worth remembering that when services go down what help desk and IT see vs. what the end user experiences may be different. When authentication services (e.g., Azure Active Directory) fail users already logged in will have a different experience to those attempting to log on afresh and many failures will have specific geographical extents. A tool like eG Enterprise allows an administrator to pinpoint what is going on if a user claims \u201cAVD is down\u201d even if they are not experiencing the issue themselves.<\/p>\n

With eG Enterprise and its best-in-class synthetic testing, administrators receive proactive alerts about not only outages but slowdowns with enough information to make decisions to avert support calls. eG Enterprise gives administrators the confidence to proactively notify users and avert calls to help and support desks.<\/p>\n

To ensure you get alerted of problems with Azure as soon as possible (and wherever you are) not only does eG Enterprise raise alerts on its alert console, but you can get alerts raised directly in your browser, sent to as SMS or email, integrated with ITSM systems<\/a> such as ServiceNow and Microsoft Teams or you can use the eG Enterprise Mobile apps for iOS or Android. The eG Enterprise Mobile apps<\/a> offer a mobile view of the monitoring system so you can see why alerts are raised, key real time and historical metrics, dashboards and more beyond simple alert messages.<\/p>\n

Or if you don\u2019t want to follow Marius\u2019 best-practice advice and externally monitor Azure services, you might want to follow him on twitter @msandbu<\/a> or LinkedIn<\/a> where he is likely to share early warnings of the next outage.<\/p>\n

\"Marius-Post\"<\/p>\n

If you are an AVD user you may like to try our free logon simulator for AVD to get proactive notifications of logon problems including Azure outages, see: Free AVD logon simulator for Azure Virtual Desktop | eG Innovations.<\/a><\/p>\n

<\/span>Further Reading:<\/span><\/h2>\n