Collect Kubernetes data

Integrate Kubernetes metrics, traces, logs, and events with Splunk Observability Cloud.

This page describes how to send Kubernetes metrics, traces, logs, and events to Splunk Observability Cloud using the Splunk Distribution of the OpenTelemetry Collector for Kubernetes.

The Splunk Distribution of the OpenTelemetry Collector provides integrated collection and forwarding for all Kubernetes telemetry. By default, the Collector for Kubernetes is deployed using a Helm chart. In a Kubernetes cluster, the chart creates a Kubernetes DaemonSet as well as other Kubernetes objects.

Prerequisites

To collect Kubernetes data, you must meet the following requirements.

  • You have the admin role in Splunk Observability Cloud.

  • You have a Splunk Enterprise license and the admin role in Splunk Enterprise.

  • You are using a supported Kubernetes environment. This Kubernetes solution has been validated in Kubernetes environments that use Minikube, Amazon Elastic Kubernetes Service (Amazon EKS), and Google Kubernetes Engine. For more information, see Supported Kubernetes distributions.

  • You have created an organization access token and confirmed that it is active. You must use an organization token to authenticate emitters that send data points to Splunk Observability Cloud.

    By default, organization access tokens are valid for one year. For access tokens created prior to February 28, 2022, the expiration date remains 5 years from the creation date. For more information, see Create and manage organization access tokens using Splunk Observability Cloud.

Collect Kubernetes data

Complete the following steps to collect Kubernetes data.

  1. Install the Collector for Kubernetes using the guided setup:

    1. From the Splunk Observability Cloud main menu, select Data Management > Available integrations.

    2. Search for and select Kubernetes.

    3. Follow the on-screen instructions to install the Collector for Kubernetes.
      Note:

      (Optional) For advanced installation instructions, see Install the Collector for Kubernetes using Helm.

  2. Set up Log Observer Connect for Splunk Enterprise.

  3. Collect logs and events with the Collector for Kubernetes.

  4. Reset the default Log Observer Connect index to point to your Kubernetes events and logs destination:

    1. Log on to Splunk Observability Cloud.

    2. Go to Settings then Log Observer connections.

      A list of your Log Observer Connect connections appears. Each connection is associated with a default Splunk platform index.

    3. Select the three-dot menu next to the connection associated with the default index you want as the default searchable index in the Log Observer Connect UI, then select Make default from the drop-down list.

      The index associated with the connection you select is the new default Splunk platform index that users can search in the Log Observer Connect UI.

Collect YAML configuration files with the Collector for Kubernetes version 0.138.1 and lower

By default, the Splunk Distribution of the OpenTelemetry Collector for Kubernetes version 0.138.1 and higher supports collecting YAML configuration files.

Pod configuration settings, statuses, and metadata (including labels and annotations) from the object configuration are displayed in YAML format on the Kubernetes entities page. For information on using these features, see Monitor Kubernetes.

To enable lower versions of the Collector for Kubernetes to collect YAML files without upgrading the Collector, complete the following steps.

  1. Add the following configuration to the clusterReceiver section in your Collector values.yaml file:
    CODE 
    k8sobjects:
        auth_type: serviceAccount
        objects:
        - interval: 6h
          mode: pull
          name: pods

    By default, YAML files are collected every 6 hours. To update the collection interval, update the interval value. For an example of this configuration in a full values.yaml file, see the clusterReceiver section in the values.yaml file of the Collector for Kubernetes GitHub repository.

  2. Restart your k8s-cluster-receiver deployment:
    CODE 
    kubectl rollout restart deployment <deployment_name>-k8s-cluster-receiver

Next steps

After you collect Kubernetes data, you can Monitor Kubernetes.

You can also export and monitor data related to your Kubernetes clusters, as described in the following table.

Get data in

Monitor

Description

Connect to the cloud service provider your Kubernetes clusters run in, if any.
Instrument back-end applications to send spans to Splunk APM

Introduction to Splunk APM

Collect metrics and spans from applications running in Kubernetes clusters.