Aggregate event data using Edge Processor

Create a data pipeline to aggregate and process event data using the pipeline editor.

1. Navigate to the Pipelines page and then select New pipeline, then Edge Processor pipeline.
2. On the Define your pipeline's partition page, do the following:
   1. Select how you want to partition your incoming data that you want to send to your pipeline.
      You can partition by source type, source, and host.
   2. Enter the conditions for your partition, including the operator and the value.
      Your pipeline will receive and process the incoming data that meets these conditions.
   3. Select Next to confirm the pipeline partition.
3. On the Add sample data page, do the following:
   1. Enter or upload sample data for generating previews that show how your pipeline processes data.
      The sample data must contain accurate examples of the values that you want to retrieve statistic example, the following sample events represent requested files from a server and contains aggregable data.

      server_name	server_ip	file_requested	bytes_out	sourcetype
      web-01	10.1.2.3	/index.html	500	web
      web-01	10.1.2.3	/images/logo.png	8500	web
      web-02	10.1.2.4	/css/main.css	1200	web

   2. Select Next to confirm the sample data that you want to use for your pipeline.
4. On the Select a data destination page, select the name of the destination that you want to send your processed data to.
5. If you selected a Splunk platform destination, you can configure index routing:
   1. Select one of the following options in the expanded destinations panel.

      Option	Description
      Default	The pipeline does not route events to a specific index. If the event metadata already specifies an index, then the event is sent to that index. Otherwise, the event is sent to the default index of the Splunk Cloud Platform deployment.
      Specify index for events with no index	The pipeline only routes events to your specified index if the event metadata did not already specify an index.
      Specify index for all events	The pipeline routes all events to your specified index.

   2. If you selected Specify index for events with no index or Specify index for events with no index, then from the Specify index for events with no index drop-down list, select the name of the index that you want to send your data to.
      Note: Be aware that the destination index is determined by a precedence order of configurations. See How does an Edge Processor know which index to send data to? for more information.

      Note: It is recommended that you do not send both non-summarized and summarized data to the same index.
6. Select Done to confirm the data destination.
   After you complete the on-screen instructions, the pipeline builder displays the SPL2 statement for your pipeline.
7. (Optional) To generate a preview of how your pipeline processes data based on the sample data that you provided, select the Preview Pipeline icon (). Use the preview results to validate your pipeline configuration.
8. Select the plus icon () in the Actions section and then select Summarize values in field.
9. In the Add summary dialog box, do the following:
   1. Add an aggregation in the Aggregations section. Select a field to aggregate and the calculation you want to perform for that field. You can add multiple fields for multiple aggregations.
      For example, using the sample data provided in Step 4a, add bytes_out as the source field and select sum as the calculation.
   2. In the Split by section, add one or more fields to group together your aggregations, if desired. Add a field to group together your aggregations, if desired.
      Continuing the example from the previous step, if you wanted to group the sum of bytes_out by the server names given in the server_name field, add server_name in the Split by section.
   The resulting output of this example would be aggregated as the following events:

   server_name	bytes_out	_raw
   web-01	9000	{"server_name": "web-01", "bytes_out": 9000}
   web_02	1200	{"server_name", "web-02", "bytes_out": 1200

   Note: Your pipeline keeps the fields specified in the Aggregations and Split by sections and filters out the unspecified fields. In the example above, the pipeline filters out the server_ip and file_requested fields because they are not included in the aggregation.
10. To save your pipeline, do the following:
    1. Select Save pipeline.
    2. In the Name field, enter a name for your pipeline.
    3. In the Description field, enter a description for your pipeline.
    4. Select Save.
11. To apply this pipeline, do the following:
    1. Navigate to the Pipelines page.
    2. In the row that lists your pipeline, select the Actions icon () and then select Apply.
    3. Select the pipeline that you want to apply, and then select Apply.
    It can take a few minutes for the process of applying the pipeline to the Edge Processor to complete. During this time, the status of the affected Edge Processors is Pending. To confirm that the process completed successfully, do the following:

    • Navigate to the Edge Processors page. Then, verify that the Instance health column for the affected Edge Processors shows that all instances are back in the Healthy status.
    • Navigate to the Pipelines page. Then, verify that the Applied column for the pipeline contains a The pipeline is applied icon ().
      Note: You might need to refresh your browser to see the latest updates.

      For information about other ways to apply pipelines to Edge Processors, see Apply pipelines to Edge Processors.

    The Edge Processors that you applied the pipeline to can now process and route data as specified in the pipeline configuration.