Kubernetes StatefulSet Pods Not Restarting After Node Drain on GKE
I'm upgrading from an older version and Quick question that's been bugging me - I'm stuck on something that should probably be simple... I've been banging my head against this for hours. I'm working with an scenario where my StatefulSet pods are not restarting automatically after I drain a node in my GKE cluster. I'm using Kubernetes v1.25 and my StatefulSet configuration looks like this: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: my-app spec: serviceName: "my-app" replicas: 3 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-app-container image: my-app-image:latest ports: - containerPort: 8080 volumeMounts: - name: my-app-storage mountPath: /data volumeClaimTemplates: - metadata: name: my-app-storage spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi ``` When I execute `kubectl drain node-name --ignore-daemonsets`, I expect the pods to be rescheduled on other nodes, but they remain in a `Pending` state, and I see the following events in the pod description: ``` Warning FailedScheduling 1m (x3 over 2m) default-scheduler 0/3 nodes are available: 3 Insufficient cpu. ``` I've tried increasing the resource requests for the StatefulSet, but it doesn't seem to help. My node pool is configured with `n1-standard-1` instances, and I've checked the quota for the project, which is sufficient for the CPU resources I need. I also ensured that the node taints are not preventing the pods from scheduling. Here's how I defined resource requests in the container spec: ```yaml resources: requests: cpu: "500m" memory: "512Mi" ``` I also checked the cluster autoscaler, but it hasn’t triggered any scale-up events despite the pending pods. Any suggestions on what could be causing this scenario or how to troubleshoot it further? I'm looking for insights on best practices for managing StatefulSets in this context. This is part of a larger API I'm building. Am I missing something obvious? Thanks in advance! For context: I'm using Yaml on Linux. Has anyone else encountered this? What's the best practice here? Thanks, I really appreciate it!