Container and Instance communication in VPC connection refused

Question

Working with the CDK (python), what I have is two containers (EC2 Services) on an EC2 Instance in NetworkMode AWS_VPC. One is supposed to be a REST backend, being reachable from the Internet, one is an internal container (DSE) doing other work, which the backend is giving it.
To not work with dynamically changing IPs, I associated cloudmap services to the EC2 Services via the cloud_map_options of each of the services, opening their ports via security_groups. As far as I understand, that means using the servicediscovery.

This works, as the DNS is actually resolving the names to the correct IPs. But the REST APIs of the Container can only be accessed inside the container itself, not from another one.

Wget on Port 8000:
```
~ $ wget dse.rcs.internal:8000
Connecting to dse.rcs.internal:8000 (10.180.3.249:8000)
wget: can't connect to remote host (10.180.3.249): Connection refused
```
Security Group (widely open for testing):
```
        self.dse_security_group = ec2.SecurityGroup(
            self,
            "dse-security-group",
            security_group_name="dse-security-group",
            vpc=self.dse_vpc,
            description="Allow access to DSE",
            allow_all_outbound=True,
            disable_inline_rules=False
        )
        self.dse_security_group.add_ingress_rule(
            peer=ec2.Peer.any_ipv4(),
            connection=ec2.Port.all_traffic()
        )
        self.dse_security_group.add_egress_rule(
            peer=ec2.Peer.any_ipv4(),
            connection=ec2.Port.all_traffic()
        )
```

Service:

```
        self.dse_task_definition = ecs.TaskDefinition(
            self,
            "dse-task-definition",
            compatibility=ecs.Compatibility.EC2,
            network_mode=ecs.NetworkMode.AWS_VPC
        )
        self.dse_container = self.dse_task_definition.add_container(
            "dse-container",
            image=ecs.ContainerImage.from_ecr_repository(
                ecr.Repository.from_repository_name(
                    self,
                    "enter",
                    repository_name="test"
                )
            ),
            memory_limit_mib=512,
            port_mappings=[
                ecs.PortMapping(
                    name="http_port_mapping",
                    container_port=8000
                )
            ],
            logging=ecs.AwsLogDriver.aws_logs(
                mode=ecs.AwsLogDriverMode.NON_BLOCKING,
                stream_prefix="DSE"
            )
        )
        self.dse_service = ecs.Ec2Service(
            self,
            "dse-service",
            service_name="dse-service",
            assign_public_ip=False,
            cluster=ecs.Cluster.from_cluster_attributes(
                self,
                "dse-cluster",
                cluster_name=ssm.StringParameter.from_string_parameter_name(self, "rcs-cluster-name", string_parameter_name="rcs-cluster-name").string_value,
                vpc=self.dse_vpc
            ),
            cloud_map_options=ecs.CloudMapOptions(
                cloud_map_namespace=self.rcs_sd_namespace,
                name="dse",
                container=self.dse_container,
                container_port=8000,
                dns_record_type=servicediscovery.DnsRecordType.A,
                dns_ttl=Duration.seconds(10)
            ),
            security_groups=[
                self.dse_security_group
            ],
            task_definition=self.dse_task_definition
        )
        
```

I also do not know how I can access an AWS_VPC network mode EC2 Service from the internet via a Loadbalancer and also access Containers in the private subnets of the VPC.

Summed up I am searching for the correct way to implement communication from:
- Internet to Loadbalancer 
- Loadbalancer to Backend Container in a VPC via DNS
- Backend Container to OtherService in the VPC via DNS

As I want to scale the container, maybe another Loadbalancer is needed as well?

Accepted Answer

Let's break down the traffic flow and steps for effective communication within your setup:

Traffic flow: Internet---> External Load Balancer ---> Backend (REST) ---> DSE

To address the "Connection refused" error while attempting to access the REST APIs on Port 8000, make sure that the task definition for your containers specifies that they are indeed listening on this port. Also, verify from the application container logs that the port is up and Listening.

The "Connection refused" error generally occurs due to incorrect IP or port settings, or the application not properly listening on the port. To diagnose further, you could try using the "dig" command to check the DNS resolution.

For proper service discovery, remember that it's primarily intended for interconnecting internal services within a VPC. When using the awsvpc network mode, you have the option to create either A records or SRV records. If you choose SRV records, ensure you're specifying the correct port that the container is listening on.

Internet to Loadbalancer :
When dealing with external access from the Internet to your load balancer, ensure the load balancer is placed in a public subnet to allow accessibility. Establish a listener and a target group for routing.

Loadbalancer to Backend Container in a VPC via DNS : 
For communication between the load balancer and the backend container in the VPC via DNS, the target group specified in the service definition ensures that, with each task launch, the container and port are registered with the target group. This enables effective routing of traffic from the load balancer to the specific container.
[+] https://docs.aws.amazon.com/AmazonECS/latest/userguide/create-application-load-balancer.html

Backend Container to other service in the VPC via DNS:
Furthermore, once the load balancer routes traffic to the REST API, the backend container's communication with DSE can be achieved using the Service Discovery DNS name (which will use the . as endpoint). Ensure that the tasks within the REST API have this Service Discovery DNS name parsed, possibly as an environment variable, to reach out to DSE effectively. You can refer to the AWS blog on Amazon ECS Service Discovery for detailed steps.
[+]  https://aws.amazon.com/blogs/aws/amazon-ecs-service-discovery/
[+]  Service discovery - Service discovery considerations - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html#service-discovery-considerations

Lastly, to address the scaling of your containers, refer to the concept of service scaling. You won't need an additional load balancer; the scaling process manages the load distribution efficiently. As the number of tasks increases, each task's container and port combination specified in the service definition is registered with the target group, streamlining the load distribution.

Container and Instance communication in VPC connection refused

Relevant content