Configure the Splunk Distribution of OTel JS for Splunk Observability Cloud

Configure the Splunk Distribution of OpenTelemetry JS to suit your instrumentation needs, like correlating traces with logs, activating exporters, and more.

You can configure the Splunk Distribution of OpenTelemetry JS to suit your instrumentation needs. In most cases, modifying the basic configuration is enough to get started.

The following sections describe all available settings for configuring OpenTelemetry for Node.js, including options for activating new features that are unique to the Splunk Distribution of OpenTelemetry JS.

Configuration methods

To configure the Splunk Distribution of OpenTelemetry JS, you can use a combination of environment variables and arguments passed to the start() function:

  • Environment variables

    For example: export OTEL_SERVICE_NAME='test-service'

  • Arguments passed to the start() function

    For example: start({ serviceName: 'my-node-service', });

Configuration for each of the supported data type, such as metrics or tracing, is set using additional properties on the configuration object:

JAVASCRIPT
start({
   // general options like `serviceName` and `endpoint`
   metrics: {
      // configuration passed to metrics signal
   },
   profiling: {
      // configuration passed to profiling signal
   },
   tracing: {
      // configuration passed to tracing signal
   },
});

You can also activate the collection of a specific data type by passing a boolean value instead of an object. For example:

JAVASCRIPT
start({
   // general options like `serviceName` and `endpoint`
   metrics: true, // turn metrics on with default options
   profiling: true, // turn profiling on with default options
});
Note: Function arguments take precedence over the corresponding environment variables.

Resource configuration

Starting with version 4.0 of the Splunk Distribution of OpenTelemetry JS, manual resource creation using new Resource({}) is no longer supported.

To define custom resource attributes, use the resourceFromAttributes() helper from @opentelemetry/resources and merge it with the detected resource:

PYTHON
import { start } from '@splunk/otel';
import { resourceFromAttributes } from '@opentelemetry/resources';

start({
  resource: (detectedResource) => {
    return detectedResource.merge(resourceFromAttributes({
      'my.attribute': 'foo',
    }));
  },
});

This approach ensures compatibility with the latest OpenTelemetry SDK and allows you to define custom resource attributes programmatically.

General settings

The following settings are specific to the Splunk Distribution of OpenTelemetry JS:

Description Environment variable Default Type Argument to start()
Log level for the OTel diagnostic logger. OTEL_LOG_LEVEL none string logLevel
Name of the service or application. OTEL_SERVICE_NAME unnamed-node-service string serviceName
Splunk authentication token. SPLUNK_ACCESS_TOKEN string accessToken
The name of your organization’s realm. SPLUNK_REALM string realm
Adds server trace info to HTTP response headers. SPLUNK_TRACE_RESPONSE_HEADER_ENABLED true boolean tracing.serverTimingEnabled

Instrumentations configuration

The following settings control which instrumentations are activated:

Description Environment variable Default Type Argument to start()
The delay in milliseconds between 2 consecutive batch span processor exports. OTEL_BSP_SCHEDULE_DELAY 5000 number
Activates all embedded instrumentations. OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED true boolean
Activates a specific instrumentation. OTEL_INSTRUMENTATION_<NAME>_ENABLED true boolean
Emit metrics from instrumentation (for example, http.server.duration). SPLUNK_INSTRUMENTATION_METRICS_ENABLED false boolean
Whether to include the full Redis query in db.statement span attributes. SPLUNK_REDIS_INCLUDE_COMMAND_ARGS false boolean
Enables GraphQL resolver spans. SPLUNK_GRAPHQL_RESOLVE_SPANS_ENABLED false boolean
Enables tracing. SPLUNK_TRACING_ENABLED true boolean

For example, to turn off all default instrumentations and only turn on the bunyan instrumentation, set the following environment variables:

SHELL
export OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false
export OTEL_INSTRUMENTATION_BUNYAN_ENABLED=true

The previous settings only apply to instrumentations loaded by the Splunk Distribution of OpenTelemetry JS by default. When using the programmatic API to supply a list of user-specified instrumentations, they have no effect.

Trace configuration

The following settings control tracing limits and attributes:

Environment variable

Argument to start()

Description

OTEL_TRACE_ENABLED

Not applicable

Activates tracer creation and autoinstrumentation. Default value is true.

OTEL_SERVICE_NAME

serviceName

Name of the service or application you’re instrumenting. Takes precedence over the service name defined in the OTEL_RESOURCE_ATTRIBUTES variable.

OTEL_RESOURCE_ATTRIBUTES

Not applicable

Comma-separated list of resource attributes added to every reported span. For example, key1=val1,key2=val2.

OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT

Not applicable

Maximum number of attributes per span. Default value is unlimited.

OTEL_SPAN_EVENT_COUNT_LIMIT

Not applicable

Maximum number of events per span. Default value is unlimited.

OTEL_SPAN_LINK_COUNT_LIMIT

Not applicable

Maximum number of links per span. Default value is 1000.

OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT

Not applicable

Maximum length of strings for attribute values. Values larger than the limit are truncated. Default value is 1200. Empty values are treated as infinity.

Samplers configuration

The following settings control trace sampling:

Environment variable

Description

OTEL_TRACES_SAMPLER

Sampler to use. The default value is parentbased_always_on. Possible values are: always_on, always_off, parentbased_always_on, parentbased_always_off, traceidratio, parentbased_traceidratio. See Built-in samplers in the official OpenTelemetry documentation for more information.

OTEL_TRACES_SAMPLER_ARG

Semicolon-separated list of rules for the rules sampler. For example, when setting the sampler to parentbased_traceidratio you can set the ratio using a number in the 0 to 1 range: OTEL_TRACES_SAMPLER_ARG=0.25.

Exporters configuration

The following settings control trace exporters and their endpoints:

Description Environment variable Default Type Argument to start()
Path to a certificate to use when verifying a server's TLS credentials. OTEL_EXPORTER_OTLP_CERTIFICATE string
Path to a certificate to use when verifying a client's TLS credentials. OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE string
Path to client's private key to use in mTLS communication in PEM format. OTEL_EXPORTER_OTLP_CLIENT_KEY string
The OTLP endpoint to export to. OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4318 string endpoint
Chooses the trace exporter protocol. Allowed values are grpc and http/protobuf. OTEL_EXPORTER_OTLP_TRACES_PROTOCOL http/protobuf string
Chooses the metric exporter protocol. Allowed values are grpc and http/protobuf. OTEL_EXPORTER_OTLP_METRICS_PROTOCOL http/protobuf string metrics.metricReaderFactory
The protocol to use for OTLP exports. OTEL_EXPORTER_OTLP_PROTOCOL http/protobuf string
The traces OTLP endpoint to export to. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT http://localhost:4318 string
Comma-separated list of metrics exporters to use. OTEL_METRICS_EXPORTER otlp string metrics.metricReaderFactory
Comma-separated list of trace exporters to use. OTEL_TRACES_EXPORTER otlp string tracing.spanExporterFactory

Jaeger exporter

To use the Jaeger exporter, add the @opentelemetry/exporter-jaeger package as in the following example:

JS
const { start } = require('@splunk/otel');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
start({
   serviceName: 'my-node-service',
   tracing: {
      spanExporterFactory: (options) => {
      return new JaegerExporter({
         serviceName: options.serviceName,
         // Additional config
      })
      }
   },
});
Note: To send data directly to Splunk Observability Cloud, see Send data directly to Splunk Observability Cloud.

Propagators configuration

The following settings control trace propagation:

Description Environment variable Default Type Argument to start()
Comma-separated list of propagators you want to use. OTEL_PROPAGATORS tracecontext,baggage string tracing.propagators

For backward compatibility with the SignalFx Tracing Library for Node.js, use the b3multi trace propagator:

Linux
SHELL
export OTEL_PROPAGATORS=b3multi
Windows PowerShell
SHELL
$env:OTEL_PROPAGATORS=b3multi

Node.js settings for AlwaysOn Profiling

The following settings control the AlwaysOn Profiling feature for the Node.js agent:

Description Environment variable Default Type Argument to start()
Activates AlwaysOn CPU profiling. SPLUNK_PROFILER_ENABLED false boolean profilingEnabled
The collector endpoint for profiler logs. SPLUNK_PROFILER_LOGS_ENDPOINT http://localhost:4318 string profiling.endpoint
Activates memory profiling for AlwaysOn Profiling. SPLUNK_PROFILER_MEMORY_ENABLED false string profiling.memoryProfilingEnabled
Frequency with which call stacks are sampled, in milliseconds. SPLUNK_PROFILER_CALL_STACK_INTERVAL 1000 number profiling.callstackInterval

To configure AlwaysOn Profiling programmatically, pass the arguments to the start function, as in the following example:

JAVASCRIPT
start({
   serviceName: '<service-name>',
   profiling: true,
   tracing: {
      // configuration passed to tracing signal
   },
});
Note: For more information on AlwaysOn Profiling, see Introduction to AlwaysOn Profiling for Splunk APM.

Metrics configuration

The following settings activate runtime metrics collection:

Description Environment variable Default Type Argument to start()
The metrics endpoint. Takes precedence over the value set in OTEL_EXPORTER_OTLP_ENDPOINT. OTEL_EXPORTER_OTLP_METRICS_ENDPOINT https://ingest.<realm>.signalfx.com/v2/datapoint/otlp number metrics.endpoint
The interval, in milliseconds, of metrics collection and exporting. OTEL_METRIC_EXPORT_INTERVAL 30000 number metrics.exportIntervalMillis
Activates metrics collection. SPLUNK_METRICS_ENABLED false boolean Activated by calling start()
The interval, in milliseconds, during which GC and event loop statistics are collected. SPLUNK_RUNTIME_METRICS_COLLECTION_INTERVAL 5000 number metrics.runtimeMetricsCollectionIntervalMillis
Activates the collection and export of runtime metrics. SPLUNK_RUNTIME_METRICS_ENABLED true boolean metrics.runtimeMetricsEnabled
Activates the collection and export of internal debug metrics for troubleshooting. SPLUNK_DEBUG_METRICS_ENABLED true boolean metrics.debugMetricsEnabled
Note: To pass settings as arguments, use the start() function.

Configuring an existing metrics client to send custom metrics

You can use an existing SignalFx client for sending custom metrics instead of creating and configuring a new one.

To configure an existing client, pass the following data to the start() function:

  • signalfx: A JavaScript object with optional client and dimensions fields. The dimensions object adds a predefined dimension for each data point. The format for dimensions is {key: value, ...}.

The following is a list of dimensions added by default:

  • service: See serviceName in Trace configuration.

  • metric_source: splunk-otel-js

  • node_version: process.versions.node, for example 16.10.0

Server trace information

To connect Real User Monitoring (RUM) requests from mobile and web applications with server trace data, activate Splunk trace response headers by setting the following environment variable:

Linux
SHELL
export SPLUNK_TRACE_RESPONSE_HEADER_ENABLED=true
Windows PowerShell
SHELL
$env:SPLUNK_TRACE_RESPONSE_HEADER_ENABLED=true

When you set this environment variable, your application instrumentation adds the following response headers to HTTP responses.

DEFAULT
Access-Control-Expose-Headers: Server-Timing
Server-Timing: traceparent;desc="00-<serverTraceId>-<serverSpanId>-01"

The Server-Timing header contains the traceId and spanId in traceparent format. For more information, see the Server-Timing and traceparent documentation on the W3C website.