If you manage GPU-intensive workloads with fluctuating traffic in a Kubernetes cluster, you can configure the Kubernetes Event-Driven Autoscaling (KEDA) component to automatically scale your GPU resources based on real-time demand. With such a setup, you’ll ensure optimal performance and cost efficiency of your application.
Gcore Managed Kubernetes includes native autoscaling based on CPU and memory only. To scale your workloads based on GPU utilization, you must manually install and configure KEDA.

​
Step 1. Create a Kubernetes cluster

Add a new Managed Kubernetes cluster with a Bare Metal worker pool:
  1. In the Gcore Customer Portal, navigate to Cloud > Kubernetes.
  2. Click Create Cluster.
  3. Select a region—the location of the data center where your cluster will be deployed.
  4. Select the Kubernetes cluster version: v.1.28.6 or higher.
  5. Configure Bare Metal pools:
  • Pool name : Enter a unique and descriptive name for your resource pool.
  • Minimum nodes and Maximum nodes : Specify how many nodes can be allocated to the pool during traffic fluctuations. The maximum number of pods must be greater than the minimum, allowing the system to scale up in response to increased demand.
  • Type : Choose Bare Metal instances.
  • Infrastructure : Select the needed GPU-optimized flavor. You need to add at least one node with GPU support to the worker pool.
Pools configuration section on the Create Kubrnetes cluster page.jpg
  1. (Optional) Add labels to include additional information about your nodes. You can also specify taints, which are special conditions that define which Kubernetes pods can run on these nodes.
  2. Disable the Autohealing nodes toggle.
  3. (Optional) Enable the Public IPv4 address option to assign public IPv4 addresses to cluster nodes.
  4. If you need to add more pools, click Add pool and configure it according to the previous steps.
A list of toggles with configurations for steps 6-9
  1. Continue configuring the cluster by following the instructions from Step 5: Select the CNI provider in the Create a Kubernetes cluster guide.

​
Step 2. Verify your Kubernetes configuration

  1. In the Gcore Customer Portal, navigate to Cloud > Kubernetes.
  2. Find the cluster you’ve created in the previous step and click its name to open it.
  3. Check the pool status—it should be Running. If you see Scaling up, wait until all resources are allocated and the cluster is ready to use.
  4. Download the .config file by clicking Kubernetes config in the top-right corner of the screen.
Kubernetes cluster overview page
  1. Export Kubernetes configuration locally:
export KUBECONFIG=/path/to/your/k8sConfig.yml
  1. Verify that the created node is ready and correctly configured by running the following command:
kubectl get nodes
Here’s an example of the expected result: 
NAME                STATUS   ROLES    AGE     VERSION 
ed-b16-82-160-248   Ready    <none>   3h26m   v1.28.6

​
Step 3. Install GPU Operator

Use Helm package manager to install the GPU Operator that’ll manage your resources. If you don’t have Helm, install it according to the official instructions. To install the GPU Operator:
  1. Add the NVIDIA Helm repository and update it as follows:
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
  1. Install the GPU Operator with the required configurations for your Kubernetes version:
  • For Kubernetes 1.28.x:
    helm install gpu-operator nvidia/gpu-operator --version v23.9.1  --wait \ 
        -n gpu-operator --create-namespace \ 
        --set driver.enabled=false \ 
        --set operator.defaultRuntime=crio \ 
        --set operator.logging.level=debug
  • For Kubernetes 1.29.x:
    helm install gpu-operator nvidia/gpu-operator --version v23.9.2  --wait \ 
        -n gpu-operator --create-namespace \ 
        --set driver.enabled=false \ 
        --set operator.defaultRuntime=crio \ 
        --set operator.logging.level=debug

​
Step 4. Verify GPU allocation

  1. Verify that GPUs have been allocated correctly. The number of GPU nodes in nvidia.com/gpu should comply with the number specified in the Bare Metal flavor.
Allocatable: 
  cpu:                192 
  ephemeral-storage:  850152999143 
  hugepages-1Gi:      0 
  hugepages-2Mi:      0 
  memory:             2113338172Ki 
  nvidia.com/gpu:     8 
  pods:               110
  1. Check the status of the Kubernetes pods–it should be READY.
kubectl get pods -n gpu-operator
Here’s an example of the expected result: 
NAME                                                         READY   STATUS      RESTARTS   AGE 
gpu-feature-discovery-f4knk                                  1/1     Running     0          3m9s 
gpu-operator-f7ffcf7f8-sf8m7                                 1/1     Running     0          3m34s 
gpu-operator-node-feature-discovery-gc-7cc7ccfff8-77g45      1/1     Running     0          3m34s 
gpu-operator-node-feature-discovery-master-d8597d549-dnxz5   1/1     Running     0          3m34s 
gpu-operator-node-feature-discovery-worker-rlf6p             1/1     Running     0          3m34s 
nvidia-container-toolkit-daemonset-rl9p2                     1/1     Running     0          3m1s 
nvidia-cuda-validator-78g72                                  0/1     Completed   0          2m32s 
nvidia-dcgm-exporter-nv9bk                                   1/1     Running     0          3m9s 
nvidia-device-plugin-daemonset-4l5fl                         1/1     Running     0          3m9s 
nvidia-mig-manager-rqnsq                                     1/1     Running     0          26s 
nvidia-operator-validator-n9rpz                              1/1     Running     0          3m9s

​
Step 5. Run test CUDA pod

Run a test GPU application to verify that the GPU resources in your Kubernetes cluster are properly configured and accessible. Follow the instructions from the official NVIDIA guide: Verification: running sample GPU application.

​
Step 6. Install KEDA for GPU autoscaling

Install and configure the Kubernetes Event-Driven Autoscaling (KEDA) component for your Managed Kubernetes cluster.
  1. Add the KEDA Helm repository:
helm repo add kedacore https://kedacore.github.io/charts
  1. Update the repository to include any new Helm charts:
helm repo update
  1. Install KEDA into its own namespace:
helm install keda kedacore/keda --namespace keda --create-namespace

​
Step 7. Install kube-prometheus-stack

To collect and visualize metrics from your Kubernetes cluster, install the kube-prometheus-stack integrated with Prometheus and Grafana.
  1. Add the Prometheus Helm repository: helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
  2. Update the repository: helm repo update
  3. Inspect the default values for the kube-prometheus-stack: helm inspect values prometheus-community/kube-prometheus-stack > values.yaml
  4. Create the values-overrides.yaml file that contains additional configurations for the GPU operator: 
serviceMonitorSelectorNilUsesHelmValues: false 

additionalScrapeConfigs: 
- job_name: gpu-metrics 
  scrape_interval: 1s 
  metrics_path: /metrics 
  scheme: http 
  kubernetes_sd_configs: 
  - role: endpoints 
    namespaces: 
      names: 
      - gpu-operator 
  relabel_configs: 
  - source_labels: [__meta_kubernetes_endpoints_name] 
    action: drop 
    regex: .*-node-feature-discovery-master 
  - source_labels: [__meta_kubernetes_pod_node_name] 
    action: replace 
    target_label: kubernetes_node
  1. Install kube-prometheus-stack with the specified configurations:
helm install prometheus-community/kube-prometheus-stack \ 
   --create-namespace --namespace prometheus \ 
   --generate-name \ 
   -f values.yaml -f values-overrides.yaml
To enable GPU metric scraping in Prometheus, you must override some default configurations and explicitly define a job that scrapes DCGM metrics from the gpu-operator namespace.

​
Step 8. Create autoscaling configuration

Define your GPU workload and configure autoscaling using KEDA with Prometheus-based GPU utilization metrics. Here is an example configuration: 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <span style="color:#FF5913">workload-name</span>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gpu
  template:
    metadata:
      labels:
        app: gpu
    spec:
      containers:
        - name: <span style="color:#FF5913">workload-name</span>
          image: <span style="color:#FF5913">app-URL</span>
          command: <span style="color:#FF5913">["--argument-1", "-argument-2"]</span>
          args: <span style="color:#FF5913">app-URL</span>
          resources:
            limits:
              nvidia.com/gpu: <span style="color:#FF5913">allocated-number-of-GPUs</span>
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gpu-dcgmproftester-deployment-scaledobject
spec:
  scaleTargetRef:
    name: <span style="color:#FF5913">workload-name</span>
  minReplicaCount: <span style="color:#FF5913">number-of-replicas</span>
  maxReplicaCount: <span style="color:#FF5913">number-of-replicas</span>
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.prometheus:9090
      metricName: <span style="color:#FF5913">metrics-name</span>
      threshold: "<span style="color:#FF5913">threshold-number</span>"
      query: sum(avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod=~"<span style="color:#FF5913">workload-name</span>.*"}[30s])) # <- HPA will divide it for number of replicas due to AVG strategy
The metric DCGM_FI_DEV_GPU_UTIL represents the real-time utilization of GPU cores. This is collected by the nvidia-dcgm-exporter, which is part of the GPU Operator setup. You may adjust the query expression or choose other metrics depending on your use case.
TipThis autoscaling configuration is based on the DCGM_FI_DEV_GPU_UTIL metrics (GPU utilization). However, you can use any of available metric from DCGM exporter.
Customize the highlighted values:
  • workload-name: Enter a unique name that identifies your workload. Make sure that you use the same value in all name fields.
  • app-URL: Specify the location of your application. For example, a Docker image.
  • allocated-number-of-GPUs: Specify how many GPUs you need. For example, 8.
  • [“/usr/bin/sample-command”]: Enter the command your container should run at startup.
  • [“—argument-1”, “-argument-2”]: Provide any arguments that should be passed to the command.
  • number-of-replicas: Provide the minimum and maximum number of workload copies the system will create during traffic fluctuation.
  • metrics-name: Enter a unique-phrase-that-distinguishes-your-metrics
  • threshold-number: If the number is reached, the system creates a new workload. For instance, “60”.

​
Step 9. Check the results

After an application with the GPU workload is deployed, Keda creates a Horizontal Pod Autoscaler (HPA) for this workload and injects the value of GPU utilization from Prometheus as an external metric.
  1. To get real-time information about the HPA, run k get hpa -w.
You should get an output like this: 
NAME                                                  REFERENCE                                  TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-gpu-dcgmproftester-deployment-scaledobject   Deployment/gpu-dcgmproftester-deployment   <unknown>/60 (avg)   1         16        0          0s
keda-hpa-gpu-dcgmproftester-deployment-scaledobject   Deployment/gpu-dcgmproftester-deployment   0/60 (avg)           1         16        1          16s
keda-hpa-gpu-dcgmproftester-deployment-scaledobject   Deployment/gpu-dcgmproftester-deployment   23/60 (avg)          1         16        1          76s
....
keda-hpa-gpu-dcgmproftester-deployment-scaledobject   Deployment/gpu-dcgmproftester-deployment   67750m/60 (avg)      1         16        8          10m
Because there are 8 GPUs per server, the system has spun up eight pods. After scaling by the number of GPUs, you’ll notice that there are some pending pods created by the HPA. The autoscaler will trigger the addition of new nodes because the cluster is out of free GPUs.
  1. Fetch the events: k get events -w.
You should get an output like this: 
....
0s          Warning   FailedScheduling      pod/gpu-dcgmproftester-deployment-9f866ff47-dnzns                             0/1 nodes are available: 1 Insufficient nvidia.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
 
....
kube-system    69s         Normal    ScaledUpGroup         configmap/cluster-autoscaler-status                                           Scale-up: setting group MachineDeployment/76-320426/gpu-autoscale-h100-h100-machine-deployment size to 2 instead of 1 (max: 2)
kube-system    69s         Normal    ScaledUpGroup         configmap/cluster-autoscaler-status                                           Scale-up: group MachineDeployment/76-320426/gpu-autoscale-h100-h100-machine-deployment size set to 2 instead of 1 (max: 2)
Wait until the node is provisioned, which might take up to 20-25 minutes.
  1. Check all pods within your cluster: k get pods.
You should see something like this: 
NAME                                            READY   STATUS    RESTARTS   AGE
apiserver-bridge-j7jk6                          1/1     Running   0          141m
gpu-dcgmproftester-deployment-9f866ff47-57zmg   1/1     Running   0          11m
gpu-dcgmproftester-deployment-9f866ff47-5wxxb   1/1     Running   0          15m
gpu-dcgmproftester-deployment-9f866ff47-8gkm8   1/1     Running   0          14m
gpu-dcgmproftester-deployment-9f866ff47-9t4bv   1/1     Running   0          7m57s
gpu-dcgmproftester-deployment-9f866ff47-dnzns   0/1     Pending   0          6m57s
gpu-dcgmproftester-deployment-9f866ff47-fb5cs   0/1     Pending   0          6m57s
gpu-dcgmproftester-deployment-9f866ff47-qthzr   1/1     Running   0          10m
gpu-dcgmproftester-deployment-9f866ff47-rxm2c   1/1     Running   0          17m
gpu-dcgmproftester-deployment-9f866ff47-tc8pq   1/1     Running   0          8m58s
gpu-dcgmproftester-deployment-9f866ff47-zbt8d   1/1     Running   0          12m
8 pods are running (there are 8 GPUs per server), and several pods have a Pending status as they wait for the provisioning of a new node. After the new node is provisioned, the system will create new pods. In the end, you’ll have the number of pods you’ve specified in the maxReplicaCount in Step 8.
InfoAfter provisioning nodes, it might take up to five minutes for the Nvidia GPU resources to be distributed and managed in the Kubernetes cluster.
  1. Check the list of nodes and related information: k get nodes.
You’ll get the list of nodes and their statuses: 
NAME                STATUS   ROLES    AGE    VERSION
ed-b16-78-161-160   Ready    <none>   166m   v1.28.6
ed-b16-78-161-169   Ready    <none>   12m    v1.28.6
  1. Double-check the HPA status: k get hpa.
You should see something like this: 
NAME                                                  REFERENCE                                  TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-gpu-dcgmproftester-deployment-scaledobject   Deployment/gpu-dcgmproftester-deployment   71463m/60 (avg)   1         16        16         41m
When the utilization metric crosses the threshold (e.g. 60%), KEDA creates new pods up to the maxReplicaCount limit. If all GPUs are in use, and there are not enough available resources, Kubernetes will keep the pods in Pending state.
Cluster Autoscaler (enabled by default in Gcore Managed Kubernetes) will then provision new nodes based on resource demand. Once the new node is ready and GPUs are initialized (which might take 5–10 minutes), the pending pods will start running.