Monitor applications in Splunk APM
Group services as applications to monitor the health, performance, and dependencies of services that support specific business capabilities.
You can use an ecommerce application to monitor the performance of ecommerce-related services and understand how they affect each other. The application can help you understand relationships and dependencies between services, such as whether downstream applications affect upstream applications.
Set a custom indexed span tag to group services as applications
By default, Splunk APM groups services into applications using the indexed span tag service.namespace. Services with the same value for this tag are grouped into the same application.
To group services using a different tag, send that tag as a span attribute on spans from each service. In OpenTelemetry, span tags are attributes. You can add them in your application instrumentation or with the Splunk Distribution of the OpenTelemetry Collector. Use the same tag key and value for services that belong to the same application, and a different value for services that belong to another application. See Add context to spans with span tags in Splunk APM.
An admin must index the tag to create Troubleshooting MetricSets before selecting it for application grouping. See Index span tags to create Troubleshooting MetricSets.
To group services by an indexed span tag:
-
From the Splunk Observability Cloud main menu, select Settings > APM services & traces.
-
Select the Applications tab.
-
Enable Group applications based on APM indexed tag.
-
Select the indexed span tag to use. Splunk APM groups services that have the same value for that tag into an application.
If you already use service.namespace in a custom metric set, that metric set continues to use its existing configuration. Review your custom metric sets before changing or removing the tag; grouping services by sf_application does not, by itself, migrate or remove custom metric set dimensions."
Monitor all applications with the Applications tab
For an aggregate monitoring view of all the applications in your environment, use the Splunk Observability Cloud main menu to select . Select the Applications tab.
The Applications tab displays a table of applications along with their health status and RED (request count, error count, and duration) metrics. Select the health status in the Health column to view the list of alerts for the application.
The health status of an application is the most critical health status of one of its services. For more information on service health statuses, see Monitor service health in the service map.
Monitor an application with the Application overview
Drill down on a specific application from the tab by selecting the name of the application in the table.
The Application overview includes a service map and a panel with information about the health, performance, and alerts of your application. The following sections describe how to use these components.
Monitor the services and relationships of an application
On the Application overview, you can use the service map to understand the services included in the application and their relationship to each other.
Select a service node from the map to display a panel with the RED metric charts.
To drill down from the panel in the service map without losing application context, select View endpoints or View instances in the panel to navigate to the Endpoints or Instances tabs in the service view with the service filter pre-populated.
To update the service map to show related applications, select the filter drop-down menu and enable the Show related applications toggle. In the map, select an application node twice to navigate to the overview for that application.
Monitor the health, performance, and alerts of an application
On the Application overview, you can monitor the health status, performance, and alerts of an application with the panel on the right side of the service map.
The Service health and Business transaction health bars show the proportion and number of services and business transactions that have Warning, Critical, or Not assessed health statuses. For streamlined health investigation, you can select a health panel to navigate to the corresponding entity tab (Services or Business transactions) in the Application overview.
To enable or learn more about health statuses, see Monitor service health in the service map and Monitor business transaction health.
The RED metric charts list the top 5 services for each metric.
The Latest alerts table shows all critical and warning alerts for the services included in the application. The icon next to the alert name reflects the alert severity. You can select an alert row to view a panel with more details about the alert, or select a Service name from the table to navigate to the service view.
To understand the severity mapped to each alert icon, use the Splunk Observability Cloud main menu to select . Reference the legend above the table.
Monitor entities related to your application
From the Application overview, you can select the Services, Business transactions, Endpoints, or Instances tab to show the list of related entities and their RED metrics. Services and business transactions also include health status. Select the name of any related entity to monitor the entity in greater detail.
Services that primarily emit client, internal, or consumer spans might have limited or empty charts in the service view. When these span kinds are included in span metrics, their spans contribute to the span count, so you can chart and alert on this activity.
This behavior depends on the span-kind metrics feature being enabled for your organization.
For more information on the Endpoints and Instances tabs, see the following sections.
Monitor endpoints related to your application
Monitor the performance of related endpoints over time and navigate to their services with the Endpoints tab.
From the Application overview, select the Endpoints tab to view a searchable table with related endpoints, the associated services, and RED (request count, error count, and duration) metrics.
By default, the Charts tab is selected and visualizes one metric in the table. Select a metric from the Sort drop-down menu to update the visualized metric. Select the Summary tab to hide the metric chart from the table and show numerical values only.
To filter the table of endpoints by a specific service, select a service from the Service drop-down menu. You can also select the name in the Service column to navigate to the service view.
Select the row for an endpoint to view a panel with the full RED metric charts, top 6 traces with errors, and top 6 longest traces. You can resize the panel by clicking and dragging the left edge.
Monitor instances related to your application
To monitor instances related to an application, complete the prerequisites for each instance you want to monitor.
If the instance is an infrastructure component, the table also displays infrastructure metrics and attributes. For infrastructure instances, the Name value is the pod name. To view the pod ID and other attributes, select the row for the instance and view the side panel.
To filter the table to show Java or .NET instances and runtime metrics, set the Service type filter to Java or .NET. You can optionally filter for a specific service with the All <service-type> filter.
Selecting a service type shows the Runtime (<service-type>) tab, which displays the heap and garbage collection metrics for the service instance in the table. You can select the APM & Infrastructure tab to view the RED and infrastructure metrics for the service instance.
In any tab, you can select the Instance ID to view the Service instance details panel. This panel includes:
-
An Overview tab with RED metric charts. Infrastructure metrics are displayed if available for the instance.
-
A Traces tab with RED metric charts, top 6 traces with errors, and top 6 longest traces. Under Traces with errors or Longest traces (> P90), select View more in Trace Analyzer to open Trace Analyzer with a filter applied for the service, service instance ID, and errors or duration.
Investigate application-level errors
When you open the Errors tab, All application errors is the default view. Select the Errors tab in the Application overview to investigate errors by error type or trace. Both views reflect the selected time range and environment.
From the Errors tab in the Application overview, select an error row to view matching traces. You can also select Traces with errors to investigate individual traces without leaving the application context.
All application errors
Select All application errors to view error types across all services in the application. For each error type, the table shows the service name and link, total error count, and a chart of error occurrences over the selected time period. The table displays the top 50 rows. Search applies only to these rows.
The exception-type chart shows application errors grouped by exception type. The HTTP-status chart shows application errors grouped by HTTP status code.
Select an error row to view a panel with matching traces, expandable stack traces, and trace context. Select View all example traces to see more traces, or select Show in trace to open the trace view with a filter applied for the error.
Traces with errors
Select Traces with errors to investigate individual traces that contain errors. The bar chart shows the rate of traces with errors over the selected time period. Traces with errors displays up to 1,000 traces.
To view traces for a specific application, filter by sf_application and select the application name. To compare traces across applications, group the results by sf_application.
The table shows each trace's timestamp, trace ID, duration, initiating operation, attributes, and services. Select View details in the Service instances column to open a popover showing the trace's errors, associated services, and affected service instances.
For traces stored in multiple segments, the service instances popover currently shows only the errors and service instances from the first stored segment.
Monitor baselines for application metrics
Use baselines to understand if a metric is behaving normally based on recent history.
| Baseline mode | Description |
|---|---|
| Daily | Understand what this metric typically looks like at this hour, averaged across the last week. |
| Weekly | Understand what this metric typically looks like at this hour, averaged across the last 4 weeks. |
Complete the following steps to add a baseline to a metric chart:
-
From the Splunk Observability Cloud main menu, select .
-
Select the Applications tab. Select an application name from the table to navigate to the application view.
-
Navigate to the tab where you want to add a baseline:
-
Select the Services or Business Transactions tab.
-
Select the Instances tab, then select an Instance ID from the table.
The selected baseline mode persists across the service view, application view, and business transaction view. Changing the baseline mode on one page updates it across all pages.Note: Baselines on the Instances tab is a controlled availability feature. In the Controlled Availability release stage, Splunk products may have limitations on customer access, features, maturity, and regional availability. For additional information on Controlled Availability, contact your Splunk representative.
-
-
Select the Baselines drop-down menu and select a baseline mode.
View the related applications for a service in the service map
You can use the service map to view the applications that interact with a service. Use this view to understand the dependencies between a service and the applications in your environment.
Complete the following steps to view the related applications for a service.
-
From the Splunk Observability Cloud main menu, select .
-
Select an application node twice.
-
Select the filter drop-down menu.
-
Enable the Show related applications toggle.