Manage your PostgreSQL database with data retention strategies

Use data retention strategies to schedule and cleanup records in your Splunk SOAR (On-premises) PostgreSQL database.

Use configure_db_maintenance to set parameters for retaining or removing database entries in the Splunk SOAR (On-premises) PostgreSQL database.

  • A set of options is called a strategy.
  • Models are collected into groups.
  • Strategies are applied to groups.

Strategy

The set of configurable parameters that define when a record should be deleted, either automatically or when the db_maintenance tool runs.

Model

Any PostgreSQL database record or Django object is called a model. Models have characteristics that define what sort of information the model represents.

Group name Description
audit Audit logs. See Enable and download audit trail logs in Splunk SOAR (On-premises).
automation_run Automation run data including Playbook Run, Action Run, App Run, Custom Function Run, and playbook run statistics.
container Containers and related records like Artifacts and Container Attachments. See About Splunk SOAR (On-premises).
indicator Indicators or Indicators of Compromise. See About Splunk SOAR (On-premises).
playbook_run_log Debug and error logs from historical playbook runs.

To use the configure_db_maintenance.py tool, follow these steps:

  1. SSH to your Splunk SOAR (On-premises) instance.
    CODE 
    SSH <username>@<SOAR_hostname>
  2. Use the following tool to manage data deletion.
    CODE 
    phenv configure_db_maintenance
  3. Add your desired argument to the data retention tool command line to list, add, edit or schedule data retention actions.
Note: On clustered systems, the configure_db_maintenance.py tool can be run from any node, but only the leader node runs the data retention strategy.

Data retention tool arguments

Add the --help argument to your command to get information on the data retention tool arguments;

CODE 
phenv configure_db_maintenance --help

Optional arguments

Use these optional arguments to manage your data retention strategy.

Argument Description
-h, --help Show this help message and exit.
-v {0,1,2,3}, --verbosity {0,1,2,3} Set the verbosity for the command's output.
  • 0 for minimal output
  • 1 for normal output
  • 2 for verbose output
  • 3 for very verbose output
--no-color Do not use color in the command output
--skip-checks Do not perform system checks.
Subcommand Description
list List all existing data retention strategies and their schedules.
add Add a new data retention strategy, or update a data retention strategy by updating all of that strategy's settings. This command has options. See the table for add later in this topic.
edit Selectively update an existing data retention strategy. This command has options. See the table for edit later in this topic.
schedule Add or update a scheduled job to service all data retention strategies. This command has options. See the table for schedule later in this topic.

configure_db_maintenance subcommands

Use configure_db_maintenance to list, add, edit or schedule data retention strategies.

List

Get a list of all data retention strategies by using the list subcommand.

CODE 
phenv configure_db_maintenance list

Sample output

PYTHON 
Enabled data retention strategies will be serviced on the following cron schedule: Every hour UTC (cron string: "0 * * * *")
Data retention strategies:
 - Group: playbook_run_log, Age to Keep: 180 days, Raw Delete: True, Enabled: True, ID: 4, Description: [SOAR System Policy] Delete stale debug and error logs from historical playbook runs

In the sample output, a data retention strategy has been scheduled to run every hour, for the playbook_run_log group, setting the Age to Keep to a value of 180 days.

Add

Apply data retention strategies to a model by using the add subcommand.

CODE 
phenv configure_db_maintenance add --help

Guidelines for adding a data retention strategy:

  • If you do not specify an audit_category or a container label, the created strategy will apply to all audit categories or container labels, if the strategy is applicable to containers or audit categories.
  • You cannot create a generic container data retention strategy (one that is for all labels) that will delete items before a label-specific strategy.
  • You cannot create a generic audit data retention strategy (one that is for all audit categories) that will delete items before a category specific strategy
Argument Required? Description
--help, -h Optional Displays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP> Required Name of the target group for the data retention strategy.
  • audit
  • automation_run
  • container
  • indicator
  • playbook_run_log
--container-label <CONTAINER LABEL> Optional Specific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels.

Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance add --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY> Optional

Specific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories.

For a list of audit categories, see the Audit categories table.

Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table. Use phenv configure_db_maintenance add --help and check the --audit-category output for the most updated list for your deployment.
--age-to-keep-time-unit <TIME UNIT> Required The time unit for measuring max-age-to-keep.
  • minutes
  • hours
  • days
  • months
  • years
--max-age-to-keep <MAX AGE TO KEEP> Required The maximum number of time units of database records to keep when applying the data retention strategy. You must set a value for --age-to-keep-time-unit <TIME UNIT>.
--description <DESCRIPTION> Optional A description of the data retention strategy.
--disabled Optional Disable the data retention strategy.
CAUTION: If you add a data retention strategy for a model that already has one, the new strategy replaces the existing strategy.

Edit

Modify data retention strategies by using the edit subcommand.

CODE 
phenv configure_db_maintenance edit --help
Argument Required? Description
--help, -h Optional Displays help for this command, then exits.
--target-group <TARGET GROUP> -t <TARGET GROUP> Required Name of the target group for the data retention strategy.
  • audit
  • automation_run
  • container
  • indicator
  • playbook_run_log

Use additional arguments when:

  • your data retention strategy targets a specific label in the container group, you must also use the --container-label argument. If you want your data retention strategy to apply to all containers, do not use the --container-label argument.
  • your data retention strategy targets specific audit items in the audit group, you must also use the --audit-category argument. If you want If you want your data retention strategy to apply to all audit items, do not use the --audit-category argument.
--container-label <CONTAINER LABEL> Optional Specific label to target if this strategy targets the container group. You can omit this argument if you want the data retention strategy to apply to all labels.

Because container labels are customizable, labels can vary by deployment. Use phenv configure_db_maintenance edit --help and check the --container-label output for the most updated list for your deployment. Check to make sure you're applying your data retention strategy to the correct label for your deployment.
--audit-category <AUDIT CATEGORY> Optional

Specific audit category to target if this strategy targets the audit_category group. You can omit this argument if you want the data retention strategy to apply to all audit categories.

For a list of audit categories, see the Audit categories table.

Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table. Use phenv configure_db_maintenance add --help and check the --audit-category output for the most updated list for your deployment.
--delete Optional Permanently delete the data retention strategy.
--enable Optional Set the selected data retention strategy to active.
--disable Optional Deactivate the data retention strategy.

Schedule

Set the schedule for data retention strategies by using the schedule subcommand.

CODE 
phenv configure_db_maintenance schedule --help
Argument Required? Description
--help, -h Optional Displays help for this command, then exits.
--cron-schedule <CRON_SCHEDULE> Required How often to run the scheduled data retention strategy. Must be a cron schedule expression.

Audit categories

Audit categories table

This table lists the available audit categories for use with the add --audit-category and edit --audit-category commands. Audit categories may change from release to release. Your deployment may have more, fewer, or different categories than listed in the Audit categories table.

Audit categories
action_run forwarder_group
aggregation_rule indicator_cef_filter
app license
app_run note
approval ph_user
artifact playbook
asset report
automation_proxy report_run
automation_rule role
automation_rule_playbook_association role_permission
container severity
container_attachment splunk_idp_role_mapping
container_comment splunk_role
container_pin system_settings
container_status tenant
custom_function workbook_phase
decided_list workbook_phase_template
es_pairing workbook_task
evidence workbook_task_template
feature_flag workbook_template

Examples

Each example is for a specific task where you may want to list, add, edit, or schedule a data retention policy.

Example: Get a list of all data existing data retention strategies

Get a list of all data retention strategies by using the list subcommand.

CODE 
phenv configure_db_maintenance list

Example: Add or update a strategy to delete indicator records

You may want to delete indicator records to improve database query performance, or to reduce the amount of disk space use by Splunk SOAR's PostgreSQL database.

Delete indicator records after three months.

CODE 
phenv configure_db_maintenance add --target-group indicator --age-to-keep-time-unit months --max-age-to-keep 3

Example: Add or update a strategy to delete audit records

You may want to delete all audit records after your company's records keeping requirements no longer need them.

Add or update a data retention strategy to delete all audit records from any category after one year:

CODE 
phenv configure_db_maintenance add --target-group audit --age-to-keep-time-unit years --max-age-to-keep 1

Example: Add or update a strategy to delete records for the app_run audit category

This example demonstrates deleting records from a single audit category after 90 days.

CODE 
phenv configure_db_maintenance add --target-group audit --audit-category app_run --age-to-keep-time-unit days --max-age-to-keep 90

Example: Turn off an existing data retention strategy

This example demonstrates tuning off a single data retention strategy that applies to all containers.

CODE 
phenv configure_db_maintenance edit --disable --target-group container

Example: Delete an existing data retention strategy

This example demonstrates how to delete a data retention strategy. This strategy applies to all containers with the events label.

CODE 
phenv configure_db_maintenance edit --delete --target-group container --container-label events

Example: Change the schedule for a data retention strategy

This example changes the schedule to run data retention strategies to every two hours, on the hour.

CODE 
phenv configure_db_maintenance schedule --cron-schedule "0 */2 * * *"