AWS Service connect between instances in a cluster

Question

Hey,

I have been running service connect between my ECS services on a single EC2 instance in bridge mode for a few months now.

Now I would like to move some services to a different EC2 instance. I would expect that nothing would change, but services that are on different instances are unable to communicate with each other.

When I exec to the docker container running on the EC2 instance, I can see that /etc/hosts contains correct service connect configuration, because its the same between all containers on both instances. But when I try to call a service in a different instance I always get *error getting response: Connection reset by peer*. The address I use with curl/wget is correct and connection will reset after a second or two. So I know I am hitting something, but I dont know what...

Service connect container logs dont contain anything helpful - in both services. Security groups are configured correctly - instances can communicate between each other,  ECS tasks can also communicate between each other and also accept traffic from itself. All services are the same namespace - I changed the instance they are deployed on. I was unable to google anything relevant for this use case.

Where else can I look for what might be wrong? Is it even possible to have multiple instance when the containers run in bridge mode?

re:Post Agent · Answer

Yes, Amazon ECS Service Connect should work between tasks running on different EC2 instances within the same cluster, even when using bridge networking mode.

Based on your description, here are some areas to check:

1. **Namespace Configuration**: Ensure all your services are in the same AWS Cloud Map namespace. Service Connect uses the namespace as a logical grouping of Amazon ECS tasks that can communicate with each other. Services can only communicate if they're in the same namespace, even if they're in different clusters.

2. **Network ACLs and Security Groups**: While you mentioned security groups are configured correctly, double-check that both the inbound and outbound rules allow traffic on the ports your services are using. Also verify any network ACLs that might be restricting traffic between subnets.

3. **Port Mappings**: Verify that your task definitions have the necessary port configurations with proper 'name' fields in the 'portMappings' section of your containers.

4. **Service Connect Configuration**: Check that Service Connect is properly enabled for all services involved. You can verify this using the AWS CLI:
```
aws ecs describe-services --cluster your-cluster-name --services your-service-name
```
Look for the "serviceConnectConfiguration" section to confirm it's enabled and properly configured.

5. **Task Connectivity**: Try testing basic network connectivity between the instances using simple tools like ping or telnet before testing the service-specific endpoints.

6. **Logs**: While you mentioned the Service Connect container logs don't show anything helpful, check the application logs of both the client and server services for any connection errors.

If you're still experiencing issues, you can use ECS Exec to connect to your tasks and run diagnostic commands:
```
aws ecs execute-command --cluster cluster-name --task task-id --container container-name --interactive --command "/bin/sh"
```

Once connected, you can use tools like curl to test connectivity to the other service and diagnose where the connection is failing.

Bridge mode networking should work across EC2 instances as long as the underlying network connectivity between the instances is properly configured and Service Connect is set up correctly in the same namespace.  
**Sources**  
[How to enable service connect to existing ECS services using AWS CLI ? | AWS re:Post](https://repost.aws/questions/QUBLpiOmSxRxSWQhHOiF_6NQ/how-to-enable-service-connect-to-existing-ecs-services-using-aws-cli)  
[Troubleshoot service connect issues in Amazon ECS | AWS re:Post](https://repost.aws/knowledge-center/ecs-troubleshoot-service-connect)  
[Use Service Connect to connect Amazon ECS services with short names - Amazon Elastic Container Service](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html)

AWS Service connect between instances in a cluster

Add your answer

Relevant content