Kubernetes probes: avoid restarts that worsen an incident

PostgreSQL struggles: latency tenfold. API pods respond in eight seconds instead of two hundred milliseconds. HTTP liveness probe times out at three seconds. Kubernetes restarts pods—lost connections, cold cache, new storm on DB. Incident becomes cascade when apps could have been temporarily removed from load balancer.

Probes are the thermometer kubelet uses to decide: send traffic, wait, or kill. Misconfigured, they turn degradation into disaster. The scenario is common because tutorials copy a single /health probe for all roles—and because liveness is configured before the team understands the difference from readiness.

Good news: a well-calibrated probe costs half a day of work. Bad news: an aggressive probe can cost hours of crashloop and amplify a database outage that might have resolved on its own.

Liveness, readiness, startup: three distinct signals

Classic mistake: one /health URL for all three roles. Under load, readiness fails and liveness kills—double penalty. Readiness should remove the pod from traffic when a dependency is slow; liveness should intervene only if the process is truly stuck, not merely slow.

Restarting a slow pod does not speed a saturated database—it adds chaos.

For an initial deployment without liveness, readiness plus Prometheus alerts often suffice. Add liveness when you clearly distinguish "slow but alive" from "dead with no recovery."

Calibrate timeouts, period, failure threshold

Three parameters determine aggressiveness:

• timeoutSeconds must exceed p99 under normal load—not ideal lab load.
• periodSeconds sets check frequency; too aggressive multiplies load in incident.
• failureThreshold × period = delay before action; three failures every ten seconds leaves thirty seconds before restart.

Example: if your API responds in 400 ms p99 under normal load, a three-second timeout leaves comfortable margin. If DB slows and p99 rises to six seconds, readiness fails—but liveness should not kill while the process still responds.

Health endpoints: separate live and ready

Separate checks in your application:

• /health/live: process responds—no heavy DB call.
• /health/ready: DB reachable, warm cache, migrations complete.

Do not put expensive DB call in liveness. If PostgreSQL is down, all pods should not crashloop—readiness suffices to remove them from Service during outage.

Configure startupProbe with thirty failures every ten seconds if boot exceeds thirty seconds: five minutes startup without premature liveness. Essential for Java apps, startup migrations, or loading a large cache.

Probes and managed European Kubernetes

On OVHcloud Managed Kubernetes, Scaleway Kapsule, or other European offers, control plane handles probes on kubelet side—you still own check content and its load.

Watch points:

• Sidecar containers adding startup latency;
• Low CPU limits causing timeouts under load;
• Confusion between external load balancer health check and pod probe—two distinct mechanisms.

Compare Kubernetes offers via the directory and comparator. On a small cluster, also see Kubernetes: do you really need it? before adding probe complexity to an oversized stack.

Observability: correlate restarts and probes

Useful metrics in incident:

• Restart counter per deployment;
• Unhealthy events in kubectl describe pod;
• Kubelet logs at restart time.

Recommended runbook: if crashloop—temporarily disable liveness after confirming process is not zombie; fix root cause (slow DB, exhausted pool); re-enable with relaxed thresholds. Never disable liveness in production without parallel availability alert.

Correlating restarts and DB latency in production often reveals an overly aggressive probe—not an application bug. See Prometheus metrics to instrument p99 before fixing timeouts.

The peak: restart as lazy reflex

A useful probe removes traffic. A lazy probe restarts and hopes the problem disappears. The difference lies in live/ready separation, timeouts calibrated on real measurements, and discipline not to copy forum config without adapting to your load.

Decide and move forward without blind spots

In half a day, you can put probes back in service of stability:

1. Implement separate live and ready endpoints—live without DB call, ready with critical dependencies.
2. Configure startupProbe if boot exceeds thirty seconds; without it, liveness kills the app before startup finishes.
3. Set timeouts from measured p99 under normal load—not a tutorial default.
4. Test in staging with simulated DB saturation: pods should go NotReady, not CrashLoopBackOff.
5. Correlate restarts and DB latency in production; adjust failureThreshold before adding replicas.
6. Compare Kubernetes offers via the directory if migrating to a managed European cluster.

Frequently asked questions

What is the difference between liveness and readiness?

Readiness removes the pod from Service without killing it—suited when a dependency is temporarily unavailable. Liveness restarts the container; reserve it for irrecoverable blocks. Confusing the two causes unnecessary restarts during transient degradation.

Why does liveness cause crashloop?

Timeouts too short, slow endpoint under load, or check sharing same saturated DB. Kubelet kills the pod when readiness would suffice. Relax thresholds or move DB check to readiness only.

What is startupProbe for?

It delays liveness and readiness during long boot—migrations, cache, remote connections. Without it, liveness kills the app before startup finishes. Configure whenever a pod takes more than thirty seconds to become operational.

HTTP or exec for a probe?

HTTP on a light app endpoint is preferred. Exec or TCP socket check little business health; reserve as last resort when HTTP is unavailable.

A useful probe removes traffic—a lazy probe restarts and hopes the problem disappears.