Kubernetes StatefulSet Pod Failures with 'Readiness Probe' Timing Out on High CPU Loads
I recently switched to I'm running a StatefulSet in a Kubernetes cluster v1.25. My application pods are configured with a readiness probe that checks an HTTP endpoint every 10 seconds, but I'm experiencing failures with the probes timing out when the CPU load spikes. The readiness probe configuration looks like this: ```yaml readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 ``` During peak load periods, my logs show errors like this: ``` Readiness probe failed: Get "http://10.128.0.5:8080/healthz": context deadline exceeded ``` I've tried increasing `timeoutSeconds` to 10 and `periodSeconds` to 20, but the question still continues. I've also monitored the CPU usage and noticed that it spikes to around 90% during these times. To mitigate this, I scaled my StatefulSet from 3 to 5 replicas, hoping it would distribute the load better, but it didn't resolve the scenario. I also checked the resource limits and requests for the containers, which are set as follows: ```yaml resources: requests: memory: "256Mi" cpu: "500m" limits: memory: "512Mi" cpu: "1000m" ``` The application is a Java Spring Boot service, and I've considered optimizing the code, but I would like to ensure the Kubernetes settings are correctly configured first. Is there a recommended approach to handling readiness probes in high load scenarios? Should I increase the resources more significantly or adjust the probe configurations further? Any insights would be greatly appreciated! I recently upgraded to Yaml 3.10. Is there a better approach?