Collect metrics and traces from a LiteLLM AI Gateway (LiteLLM Proxy)

Send metrics and traces from a LiteLLM AI Gateway (LiteLLM proxy) to the Splunk Distribution of the OpenTelemetry Collector.

You can configure the LiteLLM AI Gateway to send spans, metrics, and logs to the Splunk Distribution of the OpenTelemetry Collector.

The LiteLLM AI Gateway includes a built-in OpenTelemetry Integration module that enables an OpenTelemetry integration. The integration sends spans, metrics, and logs to the OTLP receiver.

To use this data integration, you must meet the following requirements:

  • You are using LiteLLM AI Gateway version 1.76.3 or higher. For information on LiteLLM AI Gateway versions, see the release notes in the LiteLLM GitHub repository.
    Note: LiteLLM AI Gateway is a dynamic project with complex dependencies. Upgrading your LiteLLM AI Gateway version may introduce breaking changes to your environment. Use tested versions before updating to the newest version.
  • You are using OpenTelemetry API and SDK version 1.25. Ensure that your Python packages match the following required LiteLLM AI Gateway packages:
    CODE 
    opentelemetry-api==1.25.0 
opentelemetry-sdk==1.25.0 
opentelemetry-exporter-otlp==1.25.0
  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:
  2. Add the following environment variables to the configuration of your LiteLLM AI Gateway service:
    CODE 
    LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true  
LITELLM_OTEL_INTEGRATION_ENABLE_METRICS=true  
OTEL_SERVICE_NAME=<yourServiceName>  
OTEL_EXPORTER_OTLP_PROTOCOL=<grpc, http/protobuf, or http/json> 
OTEL_EXPORTER_OTLP_ENDPOINT=<yourCollectorEndpoint>:<yourCollectorPort>  
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA
  3. Add the following code in your LiteLLM AI Gateway config.yaml file:
    CODE 
    litellm_settings:  
  callbacks: ["otel"]
  4. Run the LiteLLM AI Gateway with the startup command:
    CODE 
    'litellm --config config.yaml'

Configuration settings

Learn about the configuration settings for the OTLP receiver.

To view the configuration options for the OTLP receiver, see Settings.

Metrics

The following metrics are available for LiteLLM AI Gateways. These metrics fall under the default metric category in Splunk Observability Cloud.

For more information on these metrics, see Semantic conventions for generative AI metrics in the OpenTelemetry documentation.

Metric name Data type Metric type Unit Description
gen_ai.client.operation.duration float histogram seconds The duration of the GenAI operation.
gen_ai.client.token.usage int histogram count Measures the number of input (prompt) and output (completions) tokens used.
gen_ai.client.token.cost float histogram USD Measures the cost of the tokens used.

Attributes

The following resource attributes are available for LiteLLM AI Gateways.

Splunk Observability Cloud only displays attributes prefixed with metadata if the metadata is set in the LiteLLM administration portal or config.yaml file. For instructions, see the following pages in the LiteLLM documentation:

Attribute name Type Description
gen_ai.operation.name string

The name of the operation being performed.

Possible values:

  • chat

  • generate_content

  • text_completion
gen_ai.system string The name of the GenAI system that produced the tokens.
gen_ai.request.model string The name of the GenAI model a request is being made to.
gen_ai.framework string The name of the library used to interact with the GenAI system.
gen_ai.token.type string

The type of token being counted. For example, input or output.

This attribute is only applicable for the gen_ai.client.token.usage metric.
metadata.user_api_key_hash string The user API key as a hash value. For identifying user keys in metrics.
metadata.user_api_key_alias string The user API key alias as defined in the LiteLLM AI Gateway.
metadata.user_api_key_team_id string The ID of the team the user API key is associated with.
metadata.user_api_key_user_id string The ID of the user of the API key.
metadata.user_api_key_team_alias string The alias of the team the user API key is associated with.
metadata.requester_ip_address string The IP address of the requester, if available.
metadata.requester_metadata string Any metadata provided by LiteLLM AI Gateway about the requester.
metadata.applied_guardrails string Any guardrails applied to this request via LiteLLM AI Gateway.