Fine tuning of alert thresholds

When you are working with SAP Focused Run alert management, you might detect that some alerts are triggered too often. This will lead to work checking the alert and finding out it is more or less false alarm.

So a new threshold is needed, but what is a good threshold?

Questions that will be answered in this blog are:

  • Which tools can I use to perform fine tuning of alert thresholds?
  • Can I perform a forecast based on the data?
  • Can I perform a sensitivity analysis?
  • Which installation activities are required to enable the forecast and sensitivity functions?

Fine tuning alert threshold

In our example we will look at the Dialog Response Time metric. The current threshold for red alert is set to 5000 ms (5 seconds). The alert is triggered too often. But the question to answer now: what is a good threshold to set based on the historical data?

First click the Open metric in new window icon to enlarge the screen:

The enlarged screen now opens:

As you can see 2 times the red threshold was hit. We want to fine tune now. First select the calendar icon and select last 7 days to get full week overview:

You can use the forecast button to let the system create a forecast:

The forecast will now show mean, mean low and mean high forecast:

In this specific use case the prediction is that the maximum is 3300 ms (3.3 seconds).

Now open the statistics button to see the statistics and the recommended threshold tool button:

By changing the Sensitivity slider, the system will calculate different proposal for the alert threshold. In our case when we move sensitivity to 4 the new recommended threshold value is recalculated:

In this case it is 7669 ms.

So we now have collected following facts:

  • Current threshold of 5 seconds is reached too often
  • Average forecast based on history has a mean value of 3.3 seconds
  • Performing the sensitivity analysis the threshold recommendation is about 7.7 seconds

Based on this data the red threshold is best to increase from 5 to 8 seconds to get a good alert function. It will not reach too soon, hence limiting false alerts, but it will still alert in time in case poor performance happens.

Enabling forecast and sensitivity analysis

The forecast and sensitivity analysis function use the Application Function Library (AFL) and SAP HANA Automated Predictive Library (APL). These must be installed separately. The installation details and post steps for granting permissions are described in the Focused Run master guide in the section “Predictive Analytics Setup – Metric Forecasting”.

After the installation you must activate and assign PFCG role SAP_FRN_APP_PAS_DISP to be able to see the buttons.

Work mode management

There will be planned maintenance on SAP systems and the below databases, operating systems and the below infrastructure. During this planned maintenance, you don’t want any alerts and mails from SAP Focused Run, since the application is not available for a planned time.

Using the work mode management function you plan these maintenance events.

Questions that will be answered in this blog are:

  • How can I set up planned maintenance for systems monitored by SAP Focused Run to temporarily suppress the alerts during maintenance?
  • How can I see the current maintenance and planned upcoming maintenance?
  • Can I make a public page for the IT calendar?

Planning maintenance

Goto the IT calendar and workmode management tile:

Select the system (or select multiple systems in one go) and create the work mode for maintenance, and immediately switch to Expert Mode:

With expert mode you can simply set start and stop date and time.

You also should check the scope (the system will make a proposal) and extend if needed:

After saving, you can send notification via mail on the planned maintenance:

You can now see the planned maintenance in the calendar mode.

Overview of upcoming events

If you are in the calendar mode there can be a lot of systems. The overview screen for upcoming events will filter on the systems which have maintenance mode planned:

Actual maintenance

Actual maintenance can be seen in the System Down monitor:

You will only see maintenance that is currently running in the system down monitor, not the upcoming events.

Tips and tricks

Can I select multiple systems for planned maintenance in one go? Yes you can, this will save you a lot of time to input.

Can I make a recurring maintenance? Yes you can, in the scheduling screen mark the maintenance as recurring.

Can I make a public page for the IT calendar? Yes, you can, follow the instructions in OSS note 2926433 – IT Calendar access to Non-SAP Focused Run Users.

Alert management outbound integration to ServiceNow

SAP Focused Run alert management function can send out mails to alert to mail addresses (see this blog).

SAP Focused Run can also call an outbound integration to a ITIL tool like ServiceNow. This can help to speed up incident creation.

Questions that will be answered in this blog are:

  • How does the high level integration between SAP Focused Run and ServiceNow look like?
  • Where can I find information on the to-be-implemented ABAP BADI?
  • How can I send an alert directly to ServiceNow from the Alert management detailed page?
  • How can I automate in template settings to send an alert via outbound integration towards ServiceNow?

Setting up the integration

For setting up the integration to ServiceNow the AEM third party consumer connection BADI must be implemented. The full manual for the BADI itself can be found on the SAP Focused Run Expert portal.

The documents describes the BADI in generic way.

To call ServiceNow you have to use one of the following 2 integration methods:

  • Call webservice: in this case you import the WSDL from ServiceNow and generate the proxy and execute the SOAMANAGER settings to logon to ServiceNow. You need ABAP code in the BADI to call the proxy. See this blog for generic use of setting up webservice consumption in ABAP stack. Available webservices for ServiceNow can be found on the ServiceNow page.
  • Call the ServiceNow midserver: in this case you call a REST interface. In this case you need to setup a HTTP RFC connection to the midserver. ABAP code in the BADI is needed to make the REST call. See this blog for generic use of REST call in ABAP stack. REST API references from ServiceNow can be found on the ServiceNow page.

In a future blog we will explain the ABAP code to put inside the BAPI.

From alert trigger integration

If you are inside an alert, you can trigger the alert reaction:

Then select the reaction to forward to ServiceNow:

Within few seconds the alert in ServiceNow is created:

Alert reaction automation in template settings

The alert reaction to ServiceNow can also be automated as Outbound Integration. If you are in template maintenance mode, switch to Expert mode.

In the alerts tab now configure the alert type for Forward to and Outbound Connector:

Assign the correct variant.

If you click on the variant you go to the variant configuration screen:

Then select the outbound integration name to see the details:

Important here is the where used list, which shows you from which templates and template elements the connector is called.

Whenever the alert is raised, also the outbound integration connector to ServiceNow is called.

Self monitoring

When using Focused Run you monitor systems. But how about the health and stability of the monitoring tool itself? Here is where self monitoring plays an important role.

Questions that will be answered are:

  • What to check in self monitoring?
  • How much metrics are collected and stored in my Focused Run installation?

Self monitoring

Self monitoring can be started with the Self Monitoring FIORI tile:

If you click the tile the overview page comes:

The interesting part is unfortunately hidden in the below screen (you need to scroll), which is the CPU utilisation.

The other interesting part is the amount of data collected and stored. This is not so interesting for yourself, but more interesting for your manager to show how much data HANA can handle, or to show how much work is really automated.

Simple diagnostics agent

On the top left icons, click on the Simple DA agent button to get the agent overview screen:

Important here:

  • Check that all agents are up
  • Check that all agent versions are not too old

Monitoring and Alerting Infrastructure

The next option is to check the MAI (monitoring and alerting infrastructure) data collection:

Important here is to fix the systems in error.

Wiley Introscope

The Wiley option will show if your Wiley Introscope connected to Focused Run is ok. Wiley is used for special use cases like JAVA and Business Objects products.

Managed system overview

The managed system overview gives the overview of the diverse systems and application status:

Any red or yellow item can mean a setup issue. But it can also be because of missing authorizations and privileges of the Focused Run technical user in the connected managed system.

Central component monitoring

The central component monitoring shows the overview of the central components:

Alert management: mail notification

The alert management function is a central alert inbox function for SAP Focused Run. All alerts from all tools are coming together in the alert inbox. For full overview of functions, read this blog. This specific blog will zoom in to the option in alert management to send mails for the alert.

Questions that will be answered are:

  • How can I configure Focused Run to send mails for specific alert situations?
  • How can I setup multiple mail receivers?
  • How can I setup multiple mail groups?
  • How can I change the layout of the mail?

Setting up alert consumer

First we will set up the alert consumer. Goto the Alert Consumer Variant configuration tile:

In the next screen click on the Plus symbol to create a new Alert Consumer:

Initially there is no mail template and no recipient list.

We will create these in the steps below. When these are created, they can be used in the drop downs. Save the consumer and don’t forget to put the status to Active.

Maintain recipient list

From the alert consumer screen create a new recipient list:

Give it a name and add the e-mail addresses for the group. There can be one or multiple. Save the list.

Maintain e-mail template

Create a new e-mail template:

On the left hand side you can see the variables you can use. On the right hand side you construct the mail template. Preview is possible but shows limited functionality only. Save after you are happy with the mail.

Using the alert consumer

Now we have created the alert consumer with the mail template and recipient list. We can goto the monitoring template maintenance to assign the alert consumer. In the alerts tab of the template that you want to alert on, goto the Alerts tab:

For the type of alert switch the Automatic notification to Use Variant. In the Notifications tab below, you can now assign the created variant. Save the settings.

After the template change: do not forget to Apply and Activate the template for use.

Testing and mail sending

To test your settings: use a development system or sandbox to test your event. Then check in SOST that the mail is properly created:

Focused Run license and usage

SAP Focused Run is a licensed product. The metric is amount of GB stored in the application.

If you have more systems, more detailed metrics, with short measurement times and many functions, the more GB you will use.

Questions that will be answered in this blog are:

  • How to check the current license usage?
  • What drives the usage?
  • How can I get a cost estimate?
  • How can I create a business case for Focused Run?

Checking the license usage

In SE38 start program FRUN_USAGE_UPDATE:

Now you can see which Focused Run function uses how many MB’s.

What drives the usage?

Usage is driven by:

  • Amount of systems.
  • Details of monitoring: you can reduce the amount of items to monitor for non-productive systems. See also best practice on monitoring templates.
  • Reduce the data retention period.
  • Be careful with the RUM (Real User Management) function. This component will eat a lot of storage.

Getting a cost estimate

Your SAP account manager or the Focused Run team in Germany can give you a good cost estimate. Material number for Focused Run in the price list is 7019453.

Input for cost estimate: sizes and numbers of systems, functions of Focused Run you want to deploy, and the retention period of the data.

Output: cost estimate.

Creating the business case

The business case has 2 aspects:

  • Cost: infrastructure, license, implementation
  • Benefits

Benefits is easier to quantify if your IT service is more mature.

Elements to consider:

  • How much does an hour of outage cost on your main ECC or S4HANA core system? For lager companies, this is easily 10.000 Euro per hour or more.
  • How much does your complaint handling cost per ticket?

Benefits of Focused Run are then in avoiding half the outages by faster insights and reducing the outage costs. You cannot avoid all outages, but you can act faster.

Benefits of Focused Run are in improved clean up and issue solving. This will both reduce issues in your systems and reduce complaints and tickets you need to handle.

System analysis

System analysis is part of the Root Cause analysis functions of Focused Run. It can be used for issues analysis of current issues and for longer term trending.

Questions that will be answered in this blog are:

  • How can I start execute System Analysis for a system?
  • Which type of systems can be analysed with System Analysis?
  • How can I use the System Analysis tool for immediate analysis of issues with a system?
  • How can I use the System Analysis tool for getting insight in the longer term trends inside a system?

System analysis

Start the system analysis function by clicking on the FIORI tile for System Analysis:

Select the system you need to analyze for issues in the scope selection screen. In the first case we take an ABAP stack with time frame of the last 6 hours:

This overview might be bit overwhelming the first time. But you can see the performance was bad in the middle of the day (see top middle graph on average response time). Bottom middle graph shows CPU of some application servers was at 100%. And at the same time there were many dumps (right middle graph). This gives a clear direction were to look for issues.

The system analysis overview adjusts the information automatically to its content. This is the information for a HANA system:

Note here that the time frame here is from the last month. This is for getting longer term overview of the system behaviour. You can get this longer term overview by changing the time frame of the system analysis tool.

Page catalog

You can select a specific view from the page catalog list on the left button bar on the screen:

So you can easily filter the specific page for the type of system you need to analyze.

Guided procedure for system health check ABAP

SAP Focused Run has a guided procedure to quickly check the system health of an ABAP system.

Questions that will be answered in this blog are:

  • How to run the guided procedure for ABAP health check?
  • When to run the guided procedure for ABAP health check?

Running the ABAP health check guided procedure

To start the guided procedure for system health check, open the Guided Procedures FIORI tile:

Now select from the guided procedure for advanced system monitoring the System health check for ABAP systems:

Select the Plus icon to execute a new guided procedure:

Add the scope of systems to the guided procedure:

Then press the Execute Manually button to start.

The guided road map will now open:

Select the checks and press Perform to execute the checks. After the checks are done, you can zoom in the detailed results:

The next checks are shown below:

When to run this ABAP system health check guided procedure?

The ABAP system health check guided procedure can be run:

  1. In case of reported system issues with performance: to quickly find potential root cause
  2. For example monthly or quarterly to check how system is doing

Batch job monitoring overview

Batch job monitoring is a powerful part of SAP Focused Run system monitoring.

This blog will give you the overview of functions of batch job monitoring.

For setup of the batch job monitoring, you can read all details in this blog.

Questions that will be answered are:

  • How does SAP Focused Run job monitoring work?
  • What can I alert on?
  • Is maintenance needed for batch job monitoring?

Batch monitoring overview

For batch job monitoring start the Job Monitoring FIORI tile:

The batch job monitoring overview screen opens now:

You can now zoom into the job errors per system:

And you can zoom in per job now:

Go back to the previous screen to click on the alert generated:

During the setup of batch job monitoring (see this blog), you make the settings on alerting and who to inform when a batch job fails.

In the settings of batch job monitoring you can alert per job on:

  • Job cancellation
  • Job start delay
  • Job run time
  • Message codes in the job log

Maintenance

Maintenance on Job monitoring is required. In the initial setup the SM37 job schedule is read. But you still need to perform maintenance:

  • Jobs that no longer run need to be removed
  • Changed job behaviour (for example run time length)
  • New jobs (these you need to add)
  • New insights into job criticality and more points to monitor (you typically start with cancelled jobs only, later you learn and fine tune per job)
  • Receivers of alerts change over time

Reference

The full setup guide for batch job monitoring can be downloaded from the SAP Focused Run expert portal via this link.

Using the Alert Management function

The alert management function is a central alert inbox function for SAP Focused Run. All alerts from all tools are coming together in the alert inbox.

Questions that will be answered in this blog are:

  • How does the alert inbox work?
  • How can I get a good overview of all the alerts?
  • How can I mail an alert?
  • Which actions can I perform on an alert?
  • Can I set up my own alert dashboard?
  • Can I have Focused Run automatically confirm some of the alerts, when the system detects all is ok again?

Alert inbox

To open the Alert Inbox, click on the FIORI tile:

Don't let yourself be distracted by the high number. This is the total unfiltered amount of alerts. It will contain alerts from production and non-production systems. It will be important and non-important alerts.

Now the open alert overview dashboard will open:

There is a lot of information on this screen.

Top left are the open alerts by source. This means the open alerts by application, instance, database. In the middle top are the open alerts by category (like availability, exceptions, etc). Top right is the open alerts by current rating. Bottom left is the top type of open alert by type of metric that is causing the alert. Bottom right is the distribution of open alerts by age.

Processing an alert

From the overview you can choose two ways to start:

  1. On the top right section click on the Critical alerts that are currently still open.
  2. On the left, select the open alert list icon:

Both options will bring you to the list of open important alerts:

The sorting is done from Very High and then High, etc, already. The most important open current alerts are on top. This list can also be exported to Excel.

Clicking on an alert will open the details:

Here you can see the history and current status. It can be that the alert is till red, but it can also be that Focused Run detects that the current situation is now ok. It will still leave the alert open for you to analyse and confirm.

You can click on the Actions button to get the follow up action menu:

  • Confirm the alert will close the alert.
  • Add a comment: add text to the alert.
  • Add or change a processor: assign a user ID who should pick up the alert and is responsible for the alert.
  • Trigger an alert reaction (for example to SAP solution manager IT service desk or outbound integration to for example ServiceNow)
  • Send notification will give you the option to mail the alert:

Using the action log button:

you can see the action log for the alert:

Automatic confirmation of alerts

For some type of alerts, you might want to activate the automatic confirmation. This automatic confirmation is set at template level. Read this blog on the details. If it is set, the alert will still be created. The alert will remain open until the system detects the issue is gone. If gone, the system will automatically close the alert.

Alert management search

With the looking glass left you goto the Alert search overview. Here you can search in any way you want on the alerts, including free text search:

Top right you select extra specific filter criteria:

Alert reporting

In the left you can select the option Alert Reporting where the default alert report will open:

Clicking on any colored bar will bring you to the detailed list. From the list you can filter down to the details.

Custom alert page

By clicking on the + icon on the left button bar, you can add your own alert page:

The UI is the same as for the tactical dashboards. Read more in that blog on how to configure the dashboard.