Azure Service Principal Sign-ins Test

The Sign-in logs provided by the Azure Active Directory (AD) portal is a treasure-chest of information about user sign-ins to the Azure organization and how signed-in users use the organization's resources.

One of the four types of sign-in logs offered by Azure AD is the Service Principal sign-in log. Service principal sign-ins do not involve a user. Instead, they are sign-ins by any non-user account, such as apps or service principals (except managed identity sign-in, which are in included only in the managed identity sign-in log). In these sign-ins, the app or service provides its own credential, such as a certificate or app secret to authenticate or access resources.

If sign-in attempts of applications/service principals frequently fail, then your apps/services may be unable to access critical resources for prolonged time periods. This in turn will adversely impact app/service operations and performance. To assure your apps/services of uninterrupted access to resources and to ensure their peak performance at all times, administrators should be able to instantly detect service principal sign-in failures, investigate the reason for the failures, diagnose the root-cause, and rapidly fix it.

Sometimes, sign-in failures may not be random incidents; they could follow a definite pattern. For instance, sign-in attempts from specific IP addresses or locations may repeatedly fail. Similarly, some applications/service principals/resources may encounter more failures during sign-in than the others. Administrators should be able to detect these patterns and investigate them, as they could be hacking attempts that have to be averted in order to protect critical Azure resources.

Using the Azure Service Principal Sign-ins test, all of the above can be achieved!

This test periodically scans the messages logged in the Service Principal sign-in logs for failed sign-ins, and reports the count and details of such sign-in attempts. The granular failure metrics that the test pulls from the logs help administrators accurately identify those service principals, applications, IP addresses, locations, and resources that are seeing more sign-in failures than the rest. This way, the test sheds light on sign-in attempts that are 'suspect', so their authenticity can be verified, and any potential security risks pre-empted.

Target of the Test: A Microsoft Azure Active Directory

Agent deploying the test: A remote agent

Output of the test: One set of results for the Azure Active Directory tenant being monitored

Configurable parameters for the test
Parameters	Description
Test Period	How often should the test be executed.
Host	The host for which the test is to be configured.
Tenant ID	Specify the Directory ID of the Azure AD tenant to which the target subscription belongs. To know how to determine the Directory ID, refer to Configuring the eG Agent to Monitor Microsoft Azure Active Directory Using Microsoft Graph API
Client ID, Client Password, and Confirm Password	To connect to Azure AD, the eG agent requires an Access token in the form of an Application ID and the client secret value. If a Microsoft Azure Subscription component is already monitored in your environment, then you would have already created an Application for monitoring purposes. You can provide the Application ID and Client Secret value of that application here. However, if no such application pre-exists, you will have to create one for monitoring Azure AD. To know how to create such an application and determine its Application ID and Client Secret, refer to Configuring the eG Agent to Monitor Microsoft Azure Active Directory Using Microsoft Graph API. Specify the Application ID of the Application in the Client ID text box and the client secret value in the Client Password text box. Confirm the Client Password by retyping it in the Confirm Password text box.
Proxy Host and Proxy Port	In some environments, all communication with the Azure cloud be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none, indicating that the eG agent is not configured to communicate via a proxy, by default.
Proxy Username, Proxy Password and Confirm Password	If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy Username and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box.
Service Principal Workspace Name	Typically, the Service Principal Sign-in logs are sent to a Log Analytics Workspace. By default, the Service Principal Workspace Name parameter is set to All. his indicates that the test reads sign-in data from all Log Analytics Workspaces configured for the target tenant, by default. However, if you want the test to use only specific Log Analytics Workspaces for metrics collection, then provide the names of these workspaces here as a comma-separated list. To determine the names of the workspaces, do the following: Login to the Microsoft Azure Portal and select the Sign-in Logs option (see Figure 1). Figure 1 : Selecting the Sign-in Logs option Figure 2 will then appear listing the log entries. Next, click on the Export Data Settings button indicated by Figure 2. Figure 2 : Clicking on the Export Data Settings button The diagnostic settings that pre-exist for the sign-in logs will then appear. If any of the existing diagnostic settings have already been configured with Log Analytics Workspaces, then the Log Analytics workspace column highlighted in Figure 3 will display these workspace names. You can configure the LOG ANALYTICS WORKSPACE NAME parameter of this test with any of these workspace names. If required, you can even configure this parameter with two/more workspaces listed in Figure 3, as a comma-separated list. Figure 3 : List of Log analytics workspaces However, If the Log Analytics workspace column in Figure 3 is blank for all the existing diagnostic settings, it is a clear indication that the sign-in logs are not yet configured to be sent to any Log Analytics Workspace. In this case therefore, you should create a new diagnostic setting, where a Log Analytics Workspace is configured as the destination for the Sign-in logs. To achieve this, follow the procedure detailed in Configuring the Sign-in Logs to be Sent to a Log Analytics Workspace.
Successful Signin DD	By default, this flag is set to No. This means that, by default, this test will not report detailed diagnostics for the 'Success' metrics (eg., Successful sign-ins, Success IP addresses etc.). This is because, in a typical Azure cloud organization, there may be numerous successful sign-in attempts. A well-tuned, well-sized eG database is required for storing these detailed metrics. Without it,over a period of time, the detailed statistics of 'Successful' sign-in attempts may end up choking the eG database. To avoid it, this parameter is set to No by default. Set this flag to Yes only if your eG database has sufficient space to store detailed diagnostics for the 'Success' metrics.
No of Attempts	By default, this flag is set to 5. This means that, by default, the test will count a sign-in attempt from a specific IP address / location or for a specific application / service principal / resource as a success/failure, only if 5 or more consecutive sign-in attempts from/for that entity succeed/fail (as the case may be). For instance, if 5 or more sign-in attempts from a specific IP address fail, then the test will count that as one failure - i.e., the Failure IP addresses measure will report the value 1. Similarly, if 5 or more sign-in attempts from a specific IP address succeed, then the test will count that as a single sign-in success - i.e., the Success IP addresses measure will report the value 1. Needless to say, by default, the values of these measures will also be incremented only with every 5 or more consecutive sign-in successes/failures. This is done so that administrators are alerted only to those sign-in attempts that are 'suspect' - for example, repeated sign-in failures from the same IP address. You can change the value of this measure based on what is normal sign-in activity and what is not in your environment. This parameter applies to the following measures reported by the test: Success IP addresses Success locations Success applications Success service principals Success resource names Failure IP addresses Failure locations Failure applications Failure service principals Failure resource names
Detailed Diagnosis	To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled: The eG manager license should allow the detailed diagnosis capability Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measures made by the test:
Measurement	Description	Measurement Unit	Interpretation
Total service principal sign-in attempts	Indicates the number of sign-in attempts made by apps/service principals.	Number
Successful sign-ins	Indicates the number of sign-in attempts made by apps/service principals that were successful.	Number	Use the detailed diagnosis of this measure to know which sign-in attempts succeeded. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Failed sign-ins	Indicates the number of sign-in attempts made by apps/service principals that failed.	Number	Ideally, the value of this measure should be 0. Use the detailed diagnosis of this measure to know which sign-in attempts failed.
Success IP addresses	Indicates the number of IP addresses from which successful sign-in attempts were made.	Number	Use the detailed diagnosis of this measure to know from which IP addresses successful sign-in attempts were made. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Success locations	Indicates the number of locations from which successful sign-in attempts were made.	Number	Use the detailed diagnosis of this measure to know from which locations successful sign-in attempts were made. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Success applications	Indicates the number of applications for which sign-in attempts succeeded.	Number	Use the detailed diagnosis of this measure to know which applications signed into Azure successfully. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Success service principals	Indicates the number of service principals for which sign-in attempts succeeded.	Number	Use the detailed diagnosis of this measure to know which service principals signed into Azure successfully. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Success resource names	Indicates the number of services that were used in successful sign-ins.	Number	Use the detailed diagnosis of this measure to know which services were used in successful sign-ins. Note that this measure will report detailed diagnostics only if the Successful Signin DD parameter of this test is set to Yes.
Failed percentage	Indicates the percentage of sign-in attempts made by apps/service principals that failed.	Number	Ideally, the value of this measure should be low.
Failure IP addresses	Indicates the number of IP addresses from which sign-in attempts failed.	Number	Use the detailed diagnosis of this measure to know from which IP addresses failed sign-in attempts were made.
Failure locations	Indicates the number of locations from which sign-in attempts failed.	Number	Use the detailed diagnosis of this measure to know from which locations sign-in attempts failed.
Failure applications	Indicates the number of applications for which sign-in attempts failed.	Number	Use the detailed diagnosis of this measure to know which applications failed to sign into Azure.
Failure service principals	Indicates the number of service principals for which sign-in attempts failed.	Number	Use the detailed diagnosis of this measure to know which service principals failed to sign into Azure.
Failure resource names	Indicates the number of services that were used in failed sign-ins.	Number	Use the detailed diagnosis of this measure to know which services were used in failed sign-ins.
Not applied conditional accesses	Indicates the number of sign-ins during which no conditional access policy applied to the user and application.	Number	Conditional Access policies at their simplest are if-then statements, if a user wants to access a resource, then they must complete an action. Example: A payroll manager wants to access the payroll application and is required to do multi-factor authentication to access it.
Successful conditional accesses	Indicates the number of sign-ins during which one or more conditional access policies applied to the user and application.	Number
Failed conditional accesses	Indicates the number of sign-ins that satisfied the user and application condition of at least one Conditional Access policy and grant controls are either not satisfied or set to block access.	Number	Use the detailed diagnosis of this measure to know which conditional access policies failed.

Use the detailed diagnosis of the Failed sign-ins measure to know which sign-in attempts failed.

Figure 4 : The detailed diagnosis of the Failed sign-ins measure