Comparing Healthy and Failing Container Service Paths

Symptoms

When one containerized service works and another similar service fails, the fastest route to the fault is often a side-by-side comparison of image, runtime, service discovery, policy, and ingress state. Broad redeploys can erase the useful difference before it is understood.

Environment

Docker, Kubernetes, K3s, or similar container platforms where a healthy workload, namespace, service, or ingress path can be compared with the failing one.

Most Likely Causes

Container behavior depends on several layers that are easy to conflate: the image that actually ran, environment and secret injection, process health, readiness/liveness probes, service selectors and endpoints, cluster DNS, network policy, and ingress/TLS routing. A deployment can look present while the user-facing request still breaks at one of those boundaries.

What to Check First

• Choose a healthy comparison workload that shares the same platform and as much configuration as possible.

• Confirm the exact image digest or tag, environment/secret references, probe definitions, service selectors, endpoints, DNS name, network policy, and ingress host for both paths.

• Capture the failing request and the workload state before restarting or redeploying anything.

Related Guides

Use these when the problem moves into a neighboring part of the same workflow.

• Planning Container Runtime, Registry, and Service-Networking Failures Systematically
• Comparing Container Validation Paths for Runtime, Registry, Network, and Ingress

Operational Steps

1. Verify the two workloads are actually comparable

   Record namespace, deployment/workload name, image reference and resolved digest, replica count, node placement, environment-variable sources, secret/config references, and resource limits. A healthy workload on a different image, namespace policy, or node pool is useful context but not a clean control.

2. Compare process and probe state

   Check container start/restart counts, exit reasons, current logs, readiness, liveness, and startup probe definitions. Compare the probe path, port, timing, and observed result with the healthy workload. A Running pod that never becomes Ready is a different problem from an application that is Ready but unreachable through the service.

3. Compare service selectors and endpoints

   Confirm the Service selector matches the intended workload labels and that endpoint objects contain the expected pod addresses and ports. Compare targetPort, protocol, and named-port resolution with the working service. If the service has no endpoints, stay at selector/readiness state instead of debugging ingress.

4. Compare DNS and network policy

   Resolve the service name from an appropriate workload context and compare answers with the healthy path. Review namespace and workload network policies, egress/ingress rules, and any service-mesh policy that differs. A DNS answer only proves name resolution; test the actual destination and port from the same execution context as the failing application.

5. Compare ingress, TLS, and backend mapping

   For user-facing failures, compare ingress host/path rules, TLS secret or certificate, backend service name/port, controller events, and backend health with the healthy route. Confirm the request Host header and URL actually select the expected rule. Do not change a shared ingress controller when the failing route alone has a bad backend mapping.

6. Correct the first verified delta and retest end to end

   Change only the setting supported by the comparison—such as image digest, probe, selector, policy, service port, or ingress backend—and rerun the same request. Keep the healthy comparison unchanged so it remains a control during validation.

Validation

• The comparison identifies the first material difference between healthy and failing paths rather than producing an undifferentiated configuration diff.

• The failing workload reaches Ready state when readiness is part of the problem, and its Service exposes the expected endpoints.

• DNS and network tests succeed from the same workload context that originally failed.

• The original client request reaches the intended ingress/service/backend and returns the expected application response after the correction.

Logs to Check

• Container stdout/stderr and previous-container logs around restart or probe failures.

• Workload, pod, service, endpoint, network-policy, and ingress/controller events.

• Application logs for the exact failing request after platform reachability is proven.

Rollback and Escalation

• Preserve the original manifest or deployment revision and the healthy comparison state before changing image, probes, selectors, policy, or ingress rules.

• If the targeted change does not improve the same request path, revert that change before testing a different layer.

Escalate When

• Escalate when the comparison points to a shared cluster control such as CNI, DNS, ingress controller, service mesh, or registry that affects multiple workloads.

• Escalate when the healthy and failing workloads cannot be made meaningfully comparable without platform-owner input.

Notes from the Field

• Start with endpoints and readiness before blaming ingress. Many apparent routing problems are workloads that never became valid service backends.

• Image tags are labels; compare the resolved image digest when you need to know whether two workloads are really running the same build.