How to solve a "Connection refused" error on ECS task in awsvpc network mode?

Question

Hi there,

Even though `containerPort` as well as `hostPort` are set, we experience trouble when connecting to an ECS task from outside the container (even the host EC2 instance cannot access it). 
```
sh-4.2$ # This is the EC2 host of the task's container
sh-4.2$ curl -o /dev/null http://localhost/some/file.zip # Same with 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 80: Connection refused
```

Excerpt of the task definition:
```terraform 
network_mode          = "awsvpc" // needed in order to use A records for service discovery
network_configuration {
    subnets = [module.subnet_private.id]
}
```

Excerpt of the container definition:
```terraform
    portMappings = [
      {
        hostPort      = 80, // must equal containerPort due to awsvpc networking mode
        containerPort = 80, // see nginx.conf
        protocol      = "tcp"
      }
    ]
```

Full docker inspect:
```
[
    {
        "Id": "f852e5f1f50154f3fab574eac406fd91038a2e5514053d777d21f81c5614dc79",
        "Created": "2022-01-03T18:52:30.356339157Z",
        "Path": "/docker-entrypoint.sh",
        "Args": [
            "nginx",
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 15694,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-01-03T18:52:30.866257409Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
            "NetworkMode": "container:389dbe8d2c45cbb0ddddbbf2a8f46e62483124023880b96ef04319b7050ff5c5",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": [],
            "CapDrop": [],
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 1024,
            "Memory": 1073741824,
            "NanoCpus": 0,
            "CgroupParent": "/ecs/acafdacf06b9475b83e080cbd637f0fc",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 2147483648,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": [
                {
                    "Name": "nofile",
                    "Hard": 65536,
                    "Soft": 32768
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "Config": {
            "Hostname": "[REDACTED]",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "[REDACTED]",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.amazonaws.ecs.cluster": "Nginx_Build_agent_proxy",
                "com.amazonaws.ecs.container-name": "buildagent-proxy",
                "com.amazonaws.ecs.task-arn": "[REDACTED]",
                "com.amazonaws.ecs.task-definition-family": "buildagent-proxy",
                "com.amazonaws.ecs.task-definition-version": "20",
                "maintainer": "NGINX Docker Maintainers "
            },
            "StopSignal": "SIGQUIT"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]
```

Accepted Answer

In [`awsvpc` network mode](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking-awsvpc.html), ECS allocates a dedicated ENI for the task on the EC2 instance. From the docs:

> The task ENI is fully managed by Amazon ECS. Amazon ECS creates the ENI and attaches it to the host Amazon EC2 instance with the specified security group. The task sends and receives network traffic over the ENI in the same way that Amazon EC2 instances do with their primary network interfaces. Each task ENI is assigned a private IPv4 address by default.

Requests to `http://localhost` from a shell on the EC2 instance will not reach the container because it isn't listening on the `localhost` interface - it's listening on the dedicated ENI.

You can still make the request by using the IP address of the ENI which you can find out by:
- [Introspecting the container agent](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-introspection.html) tasks with `curl -s http://localhost:51678/v1/tasks | python -mjson.tool`
- Looking up the task in the ECS console.
- Finding the ENI used by the task in the EC2 console.

Note that you must ensure that the security group associated with that ENI allows access from source IP; that is, the primary interface associated with the EC2 instance.

Also, as mentioned in the docs, you should omit the `hostPort` field from your container definition when using `awsvpc` network mode.

How to solve a "Connection refused" error on ECS task in awsvpc network mode?

Relevant content