Addressing Service Discovery Challenges in ECS EC2 Instances with Bridge Mode

Question

We're currently exploring solutions for connecting multiple microservices within ECS EC2 instances configured in "bridge" mode.

1. First we Attempted to use service discovery with "SRV" records which proved challenging. The main challenge is SRV records and we are unable to get A record? another challenge is when we try to connect it with other service we are unable to do so? host name is not properly accepted. Not getting desired results. 
2. An alternative is leveraging service connect. The problem with service connect is: we have tcp connection in our task definition and we need to have http connection in that task definition and then we can have http connection in our service connect as opposed to tcp connection. (basically, we tried but TCP connection doesn't work)
3. Additionally, there's a potential shift to ECS Fargate with awsvpc mode and service discovery using A records. However, given specific requirements, such as the usage of RabbitMQ with multiple ports, this last option is being scrutinized for its feasibility.

We're open to insights and recommendations from the AWS community to optimize service discovery in our ECS EC2 environment. What approach would work out? we are thinking of using either 1 or 2 but so far not getting the desired results with them. we don't wish to change to ECS fargate or "awsvpc" mode instead of "bridge".

Answer

Hi Adit,
I'm trying to answer your points:
1) "*If the service task definition uses the bridge or host network mode, the SRV record is the only supported DNS record type. Create a SRV record for each service task. The SRV record must specify a container name and container port combination from the task definition.*" https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html
Moreover as Cloud Map use SRV records is not possible to directly query the service name specified, the SVR record will container the HostName and Port and TTL of the record that you will have to connect to. For this reason the application need to be able to handle this specific logic.
For example with bash script: use "nslookup" to retrive the HOSTNAME and PORT from the SRV record and then use a "curl" command to call the endpoint http://HOSTNAME:PORT

2) The Service Connect solution support ECS with bridge network mode and HTTP (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts.html) I think this would be the optimal solution if you are planning to use bridge network mode. It would be useful to have some log to double check why this solution is not working. Sometime, in my previous experiences, it could just be an issue with the creation of the endpoints. For connecting service "worker1" to service "worker2" running in a cluster with namespace "corp" you only need to use the name of the service and not the namespace: ```http://worker1 ``` instead ```http://worker1/corp```.

3) This is also a possible option, here you can see a working example https://aws.amazon.com/blogs/containers/migrate-existing-amazon-ecs-services-from-service-discovery-to-amazon-ecs-service-connect/. Service Connect will have the benefit to not rely on DNS records and this could be an important advantage because you don't need to worry about the DNS TTL and the caching layer of the DNS record at Container and Application level

Addressing Service Discovery Challenges in ECS EC2 Instances with Bridge Mode

Relevant content