>
    Enable Horizontal Pod Autoscaling on the CN-Series
    Focus
    Download PDF
    Focus
    1. Home
    2. CN-Series
    3. Deployment Modes of CN-Series Firewalls
    4. Deploy the CN-Series Firewall as a Kubernetes Service (Recommended mode of Deployment)
    5. Enable Horizontal Pod Autoscaling on the CN-Series
    Download PDF
    CN-Series

    Enable Horizontal Pod Autoscaling on the CN-Series

    Table of Contents

    Enable Horizontal Pod Autoscaling on the CN-Series

    Where Can I Use This?What Do I Need?
    • CN-Series deployment
    • CN-Series 10.1.x or above Container Images
    • Panorama running PAN-OS 10.1.x or above version
    • Helm 3.6 or above version client for CN-Series deployment with helm chart
    The horizontal pod autoscaler (HPA) is a Kubernetes resource available in all cloud environments that automatically scales the number of CN-MGMT and CN-NGFW pods in a deployment based on monitored metrics. HPA uses two standard metrics across all cloud environments—CPU and memory utilization—as well as custom metrics specific to each cloud environment. As such, each cloud requires specific yaml files to enable HPA in AKS, EKS, and GKE.
    HPA uses a cloud-specific metric adapter to retrieve metrics data from a monitoring adapter in the cloud environment, such as CloudWatch in EKS, to determine when to scale up or down based on the thresholds you define. You must modify the necessary yaml files to set the minimum and maximum number of replicas, the thresholds for each metric, and which metric are used in autoscaling your firewalls.
    In PAN OS 10.1, if you use CN-MGMT pods HPA scaling, you may scale many CN-MGMT pods with no DP pods connected to them. It is recommended to create maximum replicas of CN-MGMT pods to prevent unnecessary scaling.
    Cloud EnvironmentMetricsAverage Value
    AKS, EKS, and GKE
    CN-MGMT
    panloggingrate
    Log count
    pandataplaneslots
    Dataplane Slot count
    CN-NFGW
    dataplanecpuutilizationpct
    Percent of CN-NGFW CPU utilization
    dataplanepacketbufferutilization
    Percent of CN-NGFW packet buffer utilization
    pansessionactive
    Number of sessions active on the CN-NGFW
    pansessionutilization
    Percent of session utilization
    pansessionsslproxyutilization
    Percent of session SSL proxy utilization
    panthroughput
    Throughput in kbps
    panpacketrate
    		Packet rate in packets per second (pps)
    panconnectionspersecond
    		Connections per second
    In the example below is the pan-cn-hpa-dp.yaml file for EKS. This example is using the dataplane CPU utilization percentage to autoscale the CN-NGFW pods. At 25 percent, the cluster will scale up. If the CPU utilization reaches 50 percent, the cluster will deploy one additional pod. If the CPU utilization were to reach 75 percent, the cluster would deploy two additional pods. This is determined by dividing the total metric by the metric threshold and then deploying enough pods to bring the metric down to the configure threshold across all CN-NGFW pods in the cluster. However, the cluster will not deploy more CN-NGFW pods than the maxReplicas. If more than one metric exceed the threshold at the same time, the cluster will deploy the necessary number of pods to address the higher metric.
    By default, the HPA adapter polls the metrics adapter every 15 seconds. If the metrics you have specified exceed the configured threshold for 60 seconds, the cluster will deploy an additional CN-NGFW pod. The cluster then waits for 300 seconds (five minutes) before deciding whether or not additional CN-NGFW pods are required. By default, one pod is deployed at a time. The cluster then checks the metric (in this case, CPU utilization) after 300 seconds. If the utilization has dropped down to a level where a pod is no longer needed, the cluster will delete a pod. The cluster will then wait an additional 60 seconds before deciding if another pod can be removed.
    All the values shown below and for any metric can be modified to best fit your deployment. 
    kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
  name: hpa-dp-eks
  namespace: kube-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: pan-ngfw-dep
  minReplicas: 1
  maxReplicas: 10
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60
      - type: Percent
        value: 1
        periodSeconds: 60
      selectPolicy: Max
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 1
        periodSeconds: 300 # assuming 5 mins for dp to be ready
      - type: Percent
        value: 1
        periodSeconds: 300 # assuming 5 mins for dp to be ready
      selectPolicy: Max
  metrics:
  - type: External
    external:
      metric:
        name: dataplaneCpuUtilizationPct
      target:
        type: Value
        value: 25

    AKS

    1. Deploy an Azure Application Insights instance in your the cluster. You must provide the required Azure Application Insights Instrumentation Key and Azure Application Insight APP ID API key as a K8s secret.
    2. Download the AKS-specific HPA yaml files from the Palo Alto Networks GitHub repository.
    3. If your CN-MGMT is deployed in a custom namespace, update pan-cn-adapater.yaml with the custom namespace. The default namespace is kube-system.
    4. If you have not already done so, update the HPA parameters in the AKS-specific pan-cn-mgmt-configmap.yaml.
        #PAN_CLOUD: "AKS"
  #HPA_NAME: "<name>" #unique name to identify hpa resource per namespace or per tenant 
  #PAN_INSTRUMENTATION_KEY: "<>" #Azure APP Insight Instrumentation Key
  #PUSH_INTERVAL: "15" #time interval to publish metrics to azure app insight
    5. Edit the pan-cn-hpa-secret.yaml.
      appinsights-appid: "<Azure App Insight Application ID obtained from API Access>"
appinsights-key: "<Azure App Insight API Key created under API Access>"
azure-client-id: "<Azure SP APP ID associated with corresponding resource group with monitoring reader access>"
azure-client-secret: "<Azure SP Password associated with corresponding resource group with monitoring reader access>"
azure-tenant-id: "<Azure SP tenant ID associated with corresponding resource group with monitoring reader access>"
    6. Add the HPA name you created above to the appropriate places in pan-cn-custommetrics.yaml.
    7. Modify pan-cn-hpa-dp.yaml and pan-cn-hpa-mp.yaml.
      1. Enter the minimum and maximum number of replicas.
      2. (Optional) Change the scale down and scale up frequency values to suit your deployment. If you do not change these values, the default values are used.
      3. Copy the following section for each metric you want to use for scaling.
        - type: Pods
    pods:
      metric:
       name: pansessionactive
      target:
        type: AverageValue
averageValue: 30
      4. Change the name the metric you want to use and set the averageValue to the threshold described in the table above. If you do not change these values, the default values are used.
      5. Save you changes.
    8. Deploy the HPA yaml files. The files must be deployed in the order described below.
      1. Use Kubectl to run the pan-cn-hpa-secret.yaml
        kubectl apply -f pan-cn-hpa-secret.yaml
      2. Use Kubectl to run the pan-cn-adapter.yaml
        kubectl apply -f pan-cn-adapter.yaml
      3. Use Kubectl to run the pan-cn-custommetrics.yaml
        kubectl apply -f pan-cn-custommetrics.yaml
      4. Use Kubectl to run the pan-cn-hpa-dp.yaml
        kubectl apply -f pan-cn-hpa-dp.yaml
      5. Use Kubectl to run the pan-cn-hpa-mp.yaml
        kubectl apply -f pan-cn-hpa-mp.yaml
    9. Verify your deployment.
      • Use kubectl to verify that the custom metrics adapter pod in the custom metrics namespace.
        kubectl get pods -n custom-metrics
      • Use kubectl to check for the HPA resource.
        kubectl get hpa -n kube-system
        kubectl describe hpa <hpa-name> -n kube-system

    EKS

    1. Deploy the Amazon CloudWatch Metrics Adapter for Kubernetes in your CN-Series as a Service cluster. You must allow CloudWatch complete access to both IAM roles associated with your Kubernetes pods and clusters. To publish the custom metrics to CloudWatch, the worker nodes’ role must have the AWS managed policy CloudWatchAgentServerPolicy so that the HPA can retrieve them.
    2. Download the EKS-specific HPA yaml files from the Palo Alto Networks GitHub repository.
    3. If your CN-MGMT is deployed in a custom namespace, update pan-cn-adapater.yaml with the custom namespace. The default namespace is kube-system.
    4. Modify pan-cn-hpa-dp.yaml and pan-cn-hpa-mp.yaml.
      1. Enter the minimum and maximum number of replicas.
      2. (Optional) Change the scale down and scale up frequency values to suit your deployment. If you do not change these values, the default values are used.
      3. Copy the following section for each metric you want to use for scaling.
        - type: Pods
    pods:
      metric:
       name: pansessionactive
      target:
        type: AverageValue
averageValue: 30
      4. Change the name the metric you want to use and set the averageValue to the threshold described in the table above. If you do not change these values, the default values are used.
      5. Save you changes.
    5. Deploy the HPA yaml files. The files must be deployed in the order described below.
      1. Use Kubectl to run the pan-cn-adapter.yaml
        kubectl apply -f pan-cn-adapter.yaml
      2. Use Kubectl to run the pan-cn-externalmetrics.yaml
        kubectl apply -f pan-cn-externalmetrics.yaml
      3. Use Kubectl to run the pan-cn-hpa-dp.yaml
        kubectl apply -f pan-cn-hpa-dp.yaml
      4. Use Kubectl to run the pan-cn-hpa-mp.yaml
        kubectl apply -f pan-cn-hpa-mp.yaml
    6. Verify your deployment.
      • Use kubectl to verify that the custom metrics adapter pod in the custom metrics namespace.
        kubectl get pods -n custom-metrics
      • Use kubectl to check for the HPA resource.
        kubectl get hpa -n kube-system
        kubectl describe hpa <hpa-name> -n kube-system

    © 2024 Palo Alto Networks, Inc. All rights reserved.