Kubernetes StatefulSet Not Resuming Correctly After Node Failure in v1.30
I need some guidance on I'm deploying to production and I'm working with an scenario with a StatefulSet running in Kubernetes v1.30 where the pods do not resume correctly after a node failure... I have a StatefulSet defined with three replicas, each backed by a PersistentVolumeClaim. The goal is to ensure that when a node goes down, the pods should gracefully failover to another node and pick up where they left off. However, I noticed that after the node recovers, the pods remain in the `Pending` state and do not start up again. Here's the relevant part of my StatefulSet definition: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: my-statefulset spec: serviceName: "my-service" replicas: 3 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-container image: myimage:latest ports: - containerPort: 8080 volumeMounts: - name: my-volume mountPath: /data volumeClaimTemplates: - metadata: name: my-volume spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi ``` After simulating a node failure by cordoning and draining the node, I observed the following behavior in the `kubectl describe pod` output: ``` Warning FailedScheduling pod/my-statefulset-0 0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. ``` I've verified that the PersistentVolumeClaims are bound and available, but it seems the pod is not being scheduled on other nodes due to node selectors that I have defined for the StatefulSet. I set a node selector on the pods like this: ```yaml spec: nodeSelector: disktype: ssd ``` The nodes in my cluster have the appropriate labels, but I'm still getting the scheduling errors. I tried removing the node selector temporarily but then the pods unexpected result to attach to their respective PersistentVolumes. How can I configure my StatefulSet to handle node failures correctly while ensuring that the pods are scheduled onto the right nodes with their volumes attached? Any suggestions or insights would be really appreciated. For context: I'm using Yaml on Ubuntu 20.04. I'd love to hear your thoughts on this.