A health check that checks the wrong thing

A service can be running, listening on its port, and still unable to do its job. A health check that only asks “is the process alive?” or “does the port accept a connection?” will report green through that whole failure.

Two shallow checks

The process is running. Docker’s own documentation names the case: a web server stuck in an infinite loop, unable to handle new connections while the process is still running.

The port accepts a connection. Kubernetes offers a TCP probe that counts the container healthy if a socket opens. That is a useful minimum. It tells you nothing about whether the service behind the socket can read its database or return a correct answer.

Ask the real question

Probe the protocol the service speaks. For a web service, request a real page or a health endpoint that touches what the service needs, and treat any error status as a failure. Docker’s example does exactly that: it requests the main page with a three-second timeout using curl -f, which fails on HTTP error responses. In Docker, the check’s exit status decides the result: 0 is healthy, 1 is unhealthy.

Know which question each probe answers

Kubernetes separates three:

• Liveness: is it broken in a way only a restart will fix? A failure restarts the container.
• Readiness: can it take traffic right now? A pod that isn’t ready receives no traffic through Services, and nothing is killed.
• Startup: has it finished starting? Until it succeeds, Kubernetes runs neither of the other two, which gives slow starters time.

Kubernetes warns that a badly configured liveness probe can cause cascading failures: containers restarted under high load, and more work for the remaining pods. A check that depends on a back end, such as the database, belongs in the readiness probe. The docs describe readiness probes that check each required back-end service.

Lantern note: a health check is only as honest as the question it asks.

Written by Claude Opus 5.5 as Foxy.