Set up deployment environments in Splunk APM

Learn how to set the deployment environment tag in Splunk APM.

A deployment environment is a distinct deployment of your system or application that allows you to set up configurations that don’t overlap with configurations in other deployments of the same application. Separate deployment environments are often used for different stages of the development process, such as development, staging, and production.

A common application deployment pattern is to have multiple, distinct application environments that don’t interact directly with each other but that are all being monitored by Splunk APM: for instance, quality assurance (QA) and production environments, or multiple distinct deployments in different data centers, regions or cloud providers.

To identify the deployment environment associated with each span, you can use the deployment.environment.name or deployment.environment span tag in APM.
Note:

The deployment.environment span tag was deprecated in version 1.40.0 of the OpenTelemetry semantic conventions, but is still supported by Splunk APM. Use deployment.environment.name where possible to align with OpenTelemetry semantic conventions. If you use the deployment.environment.name span tag, it will automatically be converted to deployment.environment in APM.

Set the deployment.environment.name span tag

By setting the deployment environment as a span tag on the spans you send to Splunk APM, you can filter your APM experience by environment or environments of interest. There are two ways to add span tags to your spans: on a per-application basis during application zero-code instrumentation, or via the Splunk Distribution of OpenTelemetry Collector.

Once added to spans, the deployment.environment.name tag is automatically indexed and becomes a dimension on both Troubleshooting MetricSets and Monitoring MetricSets. To learn more about MetricSets in APM, see Learn about Troubleshooting MetricSets in APM and Learn about Monitoring MetricSets in Splunk APM.

On spans where the deployment.environment.name tag is not set, Splunk APM assumes deployment.environment.name = <unknown>.

For more information about adding span tags to spans, see Add context to spans with span tags in Splunk APM.

Note:

When you add span tags to spans through application instrumentation, you have the most control at the per-application level. The tradeoff is that it can be more time consuming to add an attribute through instrumentation for each of your applications. Adding span tags in a downstream OpenTelemetry Collector can save time, at the cost of granularity.

If a tag is applicable to 100% of the data received by the OpenTelemetry Collector in which you are adding it, adding the tag in your OpenTelemetry Collector config file is fastest. But if you need to apply logic to differentiate the spans that receive a tag through the OpenTelemetry Collector, it is likely faster to add the tags at the application level through instrumentation.

For instance, if multiple applications exist on the same host in a K8s deployment, but in different environments (for instance, production, development, and staging), setting the deployment.environment.name tag using the instrumentation library allows you to differentiate environments among spans from the same host.

Set deployment.environment.name through zero-code instrumentation

To learn how to add span tags during zero-code instrumentation, see Instrument your application code to add tags to spans.

Set deployment.environment.name in the Splunk Distribution of OpenTelemetry Collector

The Splunk Distribution of OpenTelemetry Collector agent config file includes the following resource processor for adding the deployment.environment.name tag, but it is commented out by default. Uncomment this section and add the resource/add_environment processor to a pipeline if you want to set the deployment.environment.name span tag via the Splunk Distribution of OpenTelemetry Collector instead of via instrumentation:

YAML 
resource/add_environment:
    attributes:
    action: insert
    value: staging/production/...
    key: deployment.environment.name

Note that unlike standard attributes, the deployment.environment.name tag is set with the resource processor in OpenTelemetry, because this tag is typically associated with the host or container in which the application is running.

The resource/add_environment resource can be then added to any pipeline of the service section of your configuration file. See Sample configurations for more examples.

YAML 
service:
    pipelines:
        metrics:
            processors: [resource/add_environment]
        logs:
            processors: [resource/add_environment]
        traces:
            processors: [resource/add_environment]

To learn more about how to add span tags via the Splunk Distribution of OpenTelemetry Collector, see Add span tags with the Splunk Distribution of OpenTelemetry Collector.

Alert on specific environments

You can monitor application behavior in each of your environments by setting up detectors or distinct alert thresholds for specific environments. To learn more about using detectors and alerts in APM, see Configure detectors and alerts in Splunk APM.