Configure Linux systems not running systemd

Before you can configure workload management on Linux systems not running systemd, you must create a cgroup hierarchy in which splunkd and other system processes run in their own cgroups.

The following diagram illustrates the cgroup hierarchy on Linux systems not running systemd:

The diagram shows the cgroup hierarchy on Linux machines not running under systemd. 80 percent of the total system CPU and memory is reserved for splunkd. 20 percent of the total CPU and memory is reserved for system processes.

For more information, see cgroups.

Note: Splunk Enterprise 9.4 and higher supports workload management on cgroups v2 by default.

Configure cgroups on non-systemd distributions

There are two ways to configure cgroups for workload management on Linux systems not running systemd:

Note: You must configure cpu and memory cgroups for workload management on all search heads and indexers.

Configure cgroups using cgconfig

You can use the cgconfig service installed with libcgroup to create cgroups on Linux systems not running systemd. For more information, see Using Control Groups.

To configure cgroups for workload management in Splunk Enterprise using cgconfig:

  1. Check that /sys/fs is mounted. If it is not mounted, mount the tmpfs in-memory filesystem under /sys/fs/cgroup as follows:
    CODE 
    sudo mount -t tmpfs -o size=10M tmpfs /sys/fs
    Note: Splunk Enterprise examines the /proc/mounts file to determine whether your Linux machine can support workload management.
  2. Create the cgconfig.conf file under /etc and add the following:
    CODE 
    mount {
    cpuset  = /sys/fs/cgroup/cpuset;
    cpu =     /sys/fs/cgroup/cpu;
    cpuacct = /sys/fs/cgroup/cpuacct;
    memory  = /sys/fs/cgroup/memory;
    devices = /sys/fs/cgroup/devices;
    freezer = /sys/fs/cgroup/freezer;
    net_cls = /sys/fs/cgroup/net_cls;
    blkio   = /sys/fs/cgroup/blkio;
}
 
group splunk {
    perm {
         admin {
             uid = "splunk";
             gid = "splunk";
         }
         task {
             uid = "splunk";
             gid = "splunk";
         }
     }
    cpu {
         cpu.shares = "2048";
        }
    memory {
         memory.limit_in_bytes = "3G";
           }
}
  3. Restart the cgconfig service.
    CODE 
    # service cgconfig restart
Stopping cgconfig service:                                 [  OK  ]
Starting cgconfig service:                                 [  OK  ]

    This creates the splunk cgroup.

    Note: The cgroup name must match the value of workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk.
Note: libcgroup is deprecated in RHEL 7 and 8 and has been removed from RHEL 9, as systemd now provides cgroup management for most Linux distributions.
Note: libcgroup v 3.1.0 has added systemd support to the libcgroup library, including support for cgroups v2. For more information, see https://github.com/libcgroup/libcgroup/releases. Also see Libcgroup abstraction layer.

Configure cgroups using filesystem operations

To configure cgroups, the Linux admin, logged in as root user, must create the cgroups and assign the splunk user permissions to manage the cgroups, as follows:

  1. In workload_pools.conf, set workload_pool_base_dir_name to the root cgroup to be used by splunk. For example:
    CODE 
    [general]
workload_pool_base_dir_name = splunk

    Or, send a POST request:

    CODE 
    workloads/config/set-base-dirname -workload_pool_base_dir_name <base_dir_name>

    For endpoint details, see workloads/config/set-base-dirname in the Splunk Enterprise REST API Reference Manual.

  2. Create cgroups. The cgroup path differs depending on whether you are using cgroups v1 or v2.
    For cgroups v1, create separate cpu and memory cgroups:
    CODE 
    sudo mkdir /sys/fs/cgroup/cpu/<workload_pool_base_dir_name>
sudo mkdir /sys/fs/cgroup/memory/<workload_pool_base_dir_name>

    Or, for cgroups v2, create a unified path cgroup:

    CODE 
    sudo mkdir /sys/fs/cgroup/<workload_pool_base_dir_name>
  3. Assign the splunk user permissions to manage the respective cgroups. For cgroups v1:
    CODE 
    sudo chown -R ${USER} /sys/fs/cgroup/cpu/<workload_pool_base_dir_name>
sudo chown -R ${USER} /sys/fs/cgroup/memory/<workload_pool_base_dir_name>

    Or, for cgroups v2:

    CODE 
    sudo chown -R ${USER} /sys/fs/cgroup/<workload_pool_base_dir_name>
    Note: If you have specified a user as SPLUNK_OS_USER in splunk-launch.conf, you must specify the same user as {USER} in the command. For more information see splunk-launch.conf in the Splunk Enterprise Admin Manual.
  4. Assign CPU shares and physical memory for splunk cgroup. For cgroups v1:
    CODE 
    cd /sys/fs/cgroup/cpu/splunk
echo 2048 > cpu.shares
    CODE 
    cd /sys/fs/cgroup/memory/splunk/
echo 14G > memory.limit_in_bytes

    Or, for cgroups v2:

    CODE 
    cd /sys/fs/cgroup/splunk
echo 100 > cpu.weight
    CODE 
    cd /sys/fs/cgroup/splunk/
echo 14G > memory.max

Next step

After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. See Configure workload management.