Recreate warm standby after a failover

After a failover, the previous warm standby is now a standalone primary instance of Splunk SOAR (On-premises) and the previous primary is offline or otherwise unavailable. A Splunk SOAR (On-premises) administrator can reconfigure these two instances into a new warm standby pair.

For the rest of this topic the two Splunk SOAR (On-premises) instances will be referred to as either instance A or instance B.

Instance A

The original primary Splunk SOAR (On-premises) instance.

Instance B

The original warm standby instance of Splunk SOAR (On-premises).

Configure instance B as the primary and instance A as the warm standby

This is the easiest way to reconfigure the instances for warm standby after a failover.

The initial states for your instances must be:

  • Instance A, the original primary is online but Splunk SOAR (On-premises) services are not running.
  • Instance B, the former warm standby is now a stand alone Splunk SOAR (On-premises) instance.
CAUTION: If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

Do these steps as the phantom user.

  1. SSH to instance A.
    CODE 
    SSH <username>@<instance_A_hostname>
    1. Start PostgreSQL.
      CODE 
      /<PHANTOM_HOME>/bin/phsvc start postgresql-15
    2. Start pgbouncer.
      CODE 
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
    3. Turn off primary mode
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
  2. SSH to instance B.
    CODE 
    SSH <username>@<instance_B_hostname>
    1. Configure instance B as the new primary.
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
  3. SSH to instance A.
    CODE 
    SSH <username>@<instance_A_hostname>
    1. Configure instance A as the new warm standby.
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Now the two instances are configured for warm standby. Instance B is now the primary and instance A is now the warm standby.

Configure instance A as the primary and instance B as the warm standby

This option returns the instances to the same roles they served before the failover. This can be done after you have configured the instance B as the primary using the steps in the earlier section.

CAUTION: Each time warm standby is configured the database on the standby instance is erased and the entire Splunk SOAR (On-premises) PostgreSQL database has to be streamed from the primary.

The initial states for your instances must be:

  • Instance B, the former warm standby is now a stand alone Splunk SOAR (On-premises) instance. All Splunk SOAR (On-premises) services are running.
  • Instance A, the original primary is configured as the warm standby. All Splunk SOAR (On-premises) services are running.
CAUTION: If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.
  1. SSH to instance B.
    CODE 
    SSH <username>@<instance_B_hostname>
    1. Stop Splunk SOAR (On-premises) services.
      CODE 
      /<PHANTOM_HOME>/bin/stop_phantom.sh
  2. SSH to instance A.
    CODE 
    SSH <username>@<instance_A_hostname>
    1. Configure instance A as the primary.
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --convert-to-primary
      Warm standby is disabled. Instance A is a standalone Splunk SOAR (On-premises) instance, while instance B is idle and all Splunk SOAR (On-premises) services have been shut down.
      CAUTION: If the Splunk SOAR (On-premises) instances are not in the described states, stop. Check for and do any steps which have been missed before proceeding.
  3. SSH to instance B.
    CODE 
    SSH <username>@<instance_B_hostname>
    1. Start PostgreSQL.
      CODE 
      /<PHANTOM_HOME>/bin/phsvc start postgresql-15
    2. Start pgbouncer.
      CODE 
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
    3. Turn off primary mode
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
  4. SSH to instance A.
    CODE 
    SSH <username>@<instance_A_hostname>
    1. Configure instance A as primary.
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
  5. SSH to instance B.
    CODE 
    SSH <username>@<instance_B_hostname>
    1. Configure instance B as the warm standby.
      CODE 
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Instance A and B are configured as a warm standby pair. Instance A is the primary, and instance B is the warm standby.