Search spans with Span analyzer in Splunk APM
Search, filter, and group individual spans across your applications.
Use Span analyzer to search, filter, and group individual spans across your applications to investigate latency, errors, and patterns in span data. Explore historical data to compare performance across services and span tags, or use live mode to view incoming spans in near real time.
Span analyzer can help you answer questions such as:
-
How many errors are occurring, and what are the error rates overall or for specific parameter combinations during a given time period?
-
Which services and operations have the slowest spans or highest error rates?
-
How do latency and errors vary across service versions, deployment regions, or other span tag values?
-
When do spikes in span volume, errors, or latency occur?
-
What are the minimum, maximum, and average values of selected numeric span tags?
-
How does a span with an error or high latency fit into the full request flow?
Access Span analyzer
From the Splunk Observability Cloud main menu, select . Select the Span analyzer tab.
Explore span and error counts
Use the charts above the Spans and Tags tabs to identify changes in span volume, errors, and latency over time:
-
Spans (count): Each bar shows the number of spans in a time interval. Look for spikes or drops in activity.
-
Errors (count): Each bar shows the number of spans with errors in a time interval. Compare it with the span count to investigate changes in errors alongside changes in traffic.
-
Latency (ms): The lines show P50, P90, P95, and P99 span durations in milliseconds. Compare these percentiles to understand typical latency and identify increases affecting slower spans. For example, P99 is the duration at or below which 99% of spans fall.
Explore your span data
The Spans tab is displayed by default and lists individual spans that match your selected time range and filters. The table shows each span's ID, timestamp, duration, service, operation, status, and trace ID.
In the Spans tab, you can:
-
Use the filter bar to narrow results by environment, service, duration, or span tags.
- Turn on Errors only to focus on spans with errors.
-
Select a column heading in the table to sort results, or use the settings icon to customize the columns.
-
Select a row in the table to open the Span properties side panel, which displays the span's details and metadata.
Group spans
Group spans by up to three fields to compare span volume, errors, and latency across environments, services, operations, or other available fields.
For example, you can group by Environment, then Service, then Operation to compare services within each environment and investigate specific operations.
To group your spans:
-
In the Spans tab, select Group by and select a field.
-
Select Sub grouping to add another level. You can have up to three grouping levels. Expand a group to view its subgroups. Expand the final level to view the individual spans in that group.
At each grouping level, the table displays the following metrics:
-
Count: Number of spans in the group.
-
% of total: Percentage of all matching spans in the group.
-
Errors and Error rate: Number and percentage of spans with errors in the group.
-
P99, P95, P90, and P50: Span duration percentiles for the group.
Use these metrics to identify groups with high error rates or latency, then expand them to investigate individual spans.
Examine span tag statistics
Select the Tags tab to view tag statistics. You can add or remove tags to choose which ones to examine.
| Statistic | Description |
|---|---|
| Count | Number of spans that contain the tag. |
| Coverage | Percentage of matching spans that contain the tag. |
| Unique values | Number of distinct values for the tag. |
| Most frequent | The tag value that appears most often. |
| Frequency | Number of spans containing the most frequent value. |
Select Add tag to display another tag card. Select the X on a card to remove it from the view.
Compare cards to identify common tag values, assess tag coverage, and choose attributes for further filtering or grouping.
Monitor incoming spans in live mode
Use live mode to view incoming spans and monitor changes in span volume, errors, and latency in near real time.
Turn on the Live toggle in Span analyzer to start live updates. Use the filter bar to focus on specific environments, services, durations, or span tags. You can:
-
Pause updates to inspect the displayed spans. Select a row to open the Span properties side panel and review its details and metadata.
-
Resume updates to continue viewing incoming spans.
-
Turn off live mode to return to exploring spans within a selected time range.
The Tags tab and Group by controls are unavailable in live mode. You also can't select chart bars or data points to temporarily filter results; use the filter bar instead.