Containers stop responding on public IP

Question

Hi there!

I've got a few services (Prefect and MLflow) running in ECS. These have public IPs as well as private IPs with an associated service discovery endpoint.

Access using the public IP is shaky (especially for MLflow, but both are affected). After a while, the services become unreachable using the public IP however they **still respond to requests using the private IP or service discovery endpoint**.

Spawning a new container seems to fix this, and services (usually, not always) become reachable on their new public IPs right away, but they stop working again "after a while". At this point, the services stop responding to pings as well.

I think the containers themselves are _probably_ fine, since they do respond to traffic that reach them. I'm really at a loss here, and would be grateful for any input.

// R

Accepted Answer

The issue was that containers were allowed to (re)spawn in any subnet in the VPC (I think it's random?).

Some of these had configurations which were not suitable for our services - traffic could get in, but services were not permitted to respond. Confirmed by spawning a bunch of containers and seeing which ones I can access.

The solution is to recreate the services with more carefully selected subnets.

Containers stop responding on public IP

Relevant content