Recover From a Split-Brain Condition
Focus
Focus
Advanced WildFire Powered by Precision AI™

Recover From a Split-Brain Condition

Table of Contents

Recover From a Split-Brain Condition

Where Can I Use This?What Do I Need?
  • WildFire Appliance
  • WildFire License
To resolve a split-brain condition, debug your network issues and then restore connectivity between the WildFire HA pair. WildFire appliance clusters automatically attempt to recover from split-brain conditions, but if those measures fail, you must manually initiate the recovery process.
  1. Verify that your network is operating normally and that the WildFire appliance is transmitting and receiving traffic.
    1. Enable the ability to ping on a WildFire appliance interface.
      • Enable ping on a specific appliance interface— setdeviceconfig system <interface_number> service disable-icmp no
      • Enable ping on all appliance interfaces— setdeviceconfig system service disable-icmp no
    2. Generate ping traffic from a WildFire interface to an external device. Verify that the received and transmitted counters increment.
      ping source <wildfire-interface-ip> host<destination-ip-address>
  2. Determine which WildFire appliance is unhealthy. Refer to View WildFire Cluster Status Using the CLI or View WildFire Cluster Status Using Panorama to view the status of the appliance.
  3. Gracefully restart the unhealthy node using the following command:
    request cluster reboot-local-node
    The WildFire appliance that is rebooted should auto-enroll into the WildFire cluster it was configured for.
    The remaining controller node that is in split-brain mode must be in a healthy state.
  4. Wait for the Data Migration to complete. Run show cluster data-migration-status to view the progress of the database merge. After the data merge is complete the completion timestamp displays:
    100% completed on Mon Sep 9 21:44:48 PDT 2019
    The duration of a data merge depends on the amount of data stored on the WildFire appliance. Be sure to allot at least several hours for recovery as the data merge can be a lengthy process.
  5. Verify the status of the cluster on Panorama or through the WildFire appliance CLI.