How to Set Up Dynamic Scaling for Azure Kubernetes Service with Custom Metrics?
I'm attempting to set up I'm building a feature where I've looked through the documentation and I'm still confused about I'm currently working with Azure Kubernetes Service (AKS) and trying to implement dynamic scaling based on custom metrics..... I've set up the Horizontal Pod Autoscaler (HPA) to scale my application based on CPU usage, but I want to extend it to scale based on a custom metric, specifically the number of messages in an Azure Service Bus queue. I've created a custom metrics API using Prometheus and exposed it, but I keep working with the behavior: ``` behavior from server (NotFound): the HPA was not found ``` I’ve already tried the following steps: 1. Verified the custom metrics API is reachable and returning the correct data: ``` curl http://<prometheus-service>:<port>/api/v1/query?query=queue_length ``` This returns the expected count from the Service Bus queue. 2. Configured the HPA with the custom metric: ```yaml apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: my-app-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-app minReplicas: 1 maxReplicas: 10 metrics: - type: External external: metric: name: queue_length target: type: AverageValue averageValue: 5 ``` However, I am not sure if my HPA is properly configured to reference the correct metric. 3. I also checked that the metrics server is installed and running in my cluster: ``` kubectl get pods -n kube-system | grep metrics-server ``` And it appears to be healthy. Despite these steps, my HPA still want to find the custom metrics, and I'm not clear on whether there's a configuration step I missed or if there's a specific requirement for the custom metrics API. Any guidance on how to properly set up dynamic scaling based on custom metrics in AKS would be greatly appreciated! Thanks in advance! I'm working on a web app that needs to handle this. Any ideas what could be causing this? The project is a REST API built with Yaml. Is there a simpler solution I'm overlooking? I recently upgraded to Yaml 3.11. Any help would be greatly appreciated! I'm working with Yaml in a Docker container on Ubuntu 20.04.