Troubleshoot ESXi Deployments
Focus
Focus
VM-Series

Troubleshoot ESXi Deployments

Table of Contents

Troubleshoot ESXi Deployments

Section is about troubleshooting techniques for common ESXi deployment issues, including licensing problems, connectivity failures, and interface misconfigurations.
Where Can I Use This?What Do I Need?
  • ESXi Server
  • VM-Series Firewall License (BYOL)
  • Panorama
  • VM-Series plugin
  • Panorama plugin for ESXi
Many of the troubleshooting steps for the VM-Series firewall are very similar to the hardware versions of PAN-OS. When problems occur, you should check interface counters, system log files, and if necessary, use debug to create captures.

Basic Troubleshooting

Recommendation for Network Troubleshooting Tools
It is useful to have a separate troubleshooting station to capture traffic or inject test packets in the virtualized environment. It can be helpful to build a fresh OS from scratch with common troubleshooting tools installed such as tcpdump, nmap, hping, traceroute, iperf, tcpedit, netcat, etc. This machine can then be powered down and converted to a template. Each time the tools are needed, the troubleshooting client (virtual machine) can be quickly deployed to the virtual switches in question and used to isolate networking problems. When the testing is complete, the instance can simply be discarded and the template used again the next time it is required.
For performance-related issues on the firewall, first check the Dashboard from the firewall web interface. To view alerts or create a tech support or stats dump files navigate to DeviceSupport.
For information in the vSphere client go to HomeInventoryVMs and Templates, select the VM-Series firewall instance and click the Summary tab. Under Resources, check the statistics for consumed memory, CPU, and storage. For resource history, click the Performance tab and monitor resource consumption over time.

Issues with Deploying the OVA

  • The VM-Series is delivered as a zip archive in the Open Virtualization Alliance (OVA) format that expands into three files.
    If you are having trouble deploying the OVA image, make sure the three files are unpacked and accessible. If necessary, download and extract the OVA image again.
  • The virtual disk in the OVA image is nearly 1GB. It must be present on the computer running the vSphere client, or it must be accessible as a URL for the OVA image.
    • Make sure the network connection between the vSphere client computer and the target ESXi host has low latency and sufficient bandwidth. If the connection is poor, the OVA deployment can take hours, or timeout and fail.
      You can minimize this problem if you host the image on a device in the same network as the ESXi host.
    • Any firewalls in the path must allow TCP ports 902 and 443 from the vSphere client to the ESXi host.
  • ESX 6.5.0a build 4,887,370 limits you to 2 CPU cores per socket. If you are deploying a VM-300, VM-500 or VM-700 to which you want to allocate more than 2 vCPUs per socket, refer to the VMware KB: https://kb.vmware.com/s/article/53354, for a workaround.

Why Does the Firewall Boot into Maintenance Mode?

If you have purchased the VM-1000-HV license and are deploying the VM-Series firewall in standalone mode on a VMware ESXi server, you must allocate the minimum memory your VM-Series model requires.
To avoid booting in maintenance mode, you must either modify the base image file, or edit the settings on the ESXi host or the vCenter server before you power on the VM-Series firewall.
Also, verify that the interface is VMXnet3. Setting the interface type to any other format causes the firewall to boot into maintenance mode.

Why Am I Unable to Apply for the Support or Feature License?

Have you applied the capacity auth-code on the VM-Series firewall? Before you can activate the support or feature license, you must apply the capacity auth-code so that the device can obtain a serial number. This serial number is required to activate the other licenses on the VM-Series firewall.

Why Does My Cloned VM-Series Firewall Not Have a Valid License?

VMware assigns a unique UUID to each virtual machine including the VM-Series firewall. So, when a VM-Series firewall is cloned, a new UUID is assigned to it. Because the serial number and license for each instance of the VM-Series firewall is tied to the UUID, cloning a licensed VM-Series firewall results in a new firewall with an invalid license. You need a new auth-code to activate the license on the newly deployed firewall. Apply the capacity auth-code and a new support license to obtain full functionality, support, and software upgrades on the VM-Series firewall.

Does Moving the VM-Series Firewall Cause License Invalidation?

If you are manually moving the VM-Series firewall from one host to another, be sure to select the option, This guest was moved to prevent license invalidation.

Why Is the VM-Series Firewall Not Receiving Any Network Traffic?

On the VM-Series firewall, check the traffic logs (MonitorLogs). If the logs are empty, use the following CLI command to view the packets on the interfaces of the VM-Series firewall:
show counter global filter
delta yes 
Global counters: 
Elapsed time since last sampling: 594.544 seconds 
-------------------------------------------------------------------------------- 
Total counters shown: 0 
-------------------------------------------------------------------------------- 
In the vSphere environment, check for the following issues:
  • Check the port groups and confirm that the firewall and the virtual machines are on the correct port group
    Make sure that the interfaces are mapped correctly.
    Network adapter 1 = management
    Network adapter 2= Ethernet1/1
    Network adapter 3 = Ethernet1/2
    For each virtual machine, check the settings to verify the interface is mapped to the correct port group.
  • Verify that either promiscuous mode is enabled for each port group or for the entire switch or that you have configured the firewall to Hypervisor Assigned MAC Addresses.
    Since the dataplane PAN-OS MAC addresses are different from the vNIC MAC addresses assigned by vSphere, the port group (or the entire vSwitch) must be in promiscuous mode if not enabled to use the hypervisor assigned MAC address:
    • Check the VLAN settings on vSphere.
      The use of the VLAN setting for the vSphere port group serves two purposes: It determines which port groups share a layer 2 domain, and it determines whether the uplink ports are tagged (802.1Q).
    • Check the physical switch port settings
      If a VLAN ID is specified on a port group with uplink ports, then vSphere uses 802.1Q to tag outbound frames. The tag must match the configuration on the physical switch or the traffic does not pass.
      Check the port statistics if using virtual distributed switches (vDS); Standard switches do not provide any port statistics