- Newest
- Most votes
- Most comments
Hi,
Standard DNS traffic is on port 53. Most often it works on UDP but in rare occasions, it can be TCP as well.
So, check that the security groups (plural because it can be at individual task level, at cluster or at VPC level) related to your container all allow outbound trafic on port 53 for UDP and TCP.
Best,
Didier
Hi Didier, Thanks for the response. the tasks are in public subnet with outgoing traffic is open. I am not sure why it only happens with this particular domain.
Also I just did nslookup within the task container and I am able to see the response below.
/usr/src/app # nslookup webpay3gint.transbank.cl Server: 8.8.8.8 Address: 8.8.8.8:53
Non-authoritative answer: webpay3gint.transbank.cl canonical name = webpay3gint.tbk.akadns.net Name: webpay3gint.tbk.akadns.net Address: 200.10.12.94
Non-authoritative answer: webpay3gint.transbank.cl canonical name = webpay3gint.tbk.akadns.net
So something has changed, I added a dhcp option set with Google DNS for the above domain and as last night nslookup was working in task container, now I have removed it to use AmazonDNS, below is the response of nslookup.
nslookup webpay3gint.transbank.cl Server: 10.0.0.2 Address: 10.0.0.2:53
** server can't find webpay3gint.transbank.cl: NXDOMAIN
** server can't find webpay3gint.transbank.cl: NXDOMAIN
If you can deduce something from it.
Hi,
It looks like issue with DNS, to test it create another ECS Task and use another DNS. Keep checking the cloud watch log and cloud trail.
Thanks
Thanks for the answer, but other domain names are working fine, its just this domain.
Also I just did nslookup within the task container and I am able to see the response below.
/usr/src/app # nslookup webpay3gint.transbank.cl Server: 8.8.8.8 Address: 8.8.8.8:53
Non-authoritative answer: webpay3gint.transbank.cl canonical name = webpay3gint.tbk.akadns.net Name: webpay3gint.tbk.akadns.net Address: 200.10.12.94
Non-authoritative answer: webpay3gint.transbank.cl canonical name = webpay3gint.tbk.akadns.net
Do you have any route53 firewalls setup? This could be blocking the domain
I can confirm that there are no route53 firewalls setup. and that domain is an external payment gateway
The issue you're experiencing with DNS resolution in AWS ECS Fargate for a specific domain while others work correctly could be due to several factors. Here are some potential reasons and solutions to consider:
-
DNS Configuration: Ensure that your ECS Fargate task is using the correct DNS server. By default, it should be using AmazonProvidedDNS. If you're using custom domain name servers in your VPC instead of AmazonProvidedDNS, this could cause resolution issues for certain domains.
-
VPC DNS Settings: Check if the VPC DNS Hostnames option is enabled. If it's turned off, this can lead to DNS resolution problems.
-
Network Configuration: Since you mentioned the task is in a public subnet, verify that the task has a public IP address assigned and is properly connected to an Internet Gateway (IGW). This is necessary for outbound internet access, which is required for DNS resolution of public domains.
-
Security Groups and NACLs: Review your security group rules and network ACLs to ensure they're not blocking outbound DNS traffic (usually on port 53).
-
Route Configuration: Confirm that your VPC has the correct route table entries to allow traffic to flow to the internet via the IGW.
-
Task IAM Role: Ensure that the IAM role associated with your ECS task has the necessary permissions to make outbound connections.
-
Firewall or DNS Filtering: Check if there are any firewalls or DNS filtering mechanisms in place that might be blocking or interfering with requests to this specific domain.
Since replacing the hostname with the IP address works, it suggests that the issue is specifically related to DNS resolution rather than connectivity. This points towards a problem with either the DNS configuration or how the Fargate task is resolving DNS queries.
To troubleshoot further, you could try the following:
- Use the AWS CLI or console to verify the VPC and subnet settings.
- Check the ECS task definition and ensure it's configured to use the appropriate networking mode.
- Try running a similar task on EC2 to see if the issue persists, which could help isolate whether it's specific to Fargate or a broader networking issue.
If the problem continues after checking these areas, you may need to contact AWS support for more in-depth troubleshooting of your specific environment configuration.
Sources
Troubleshoot ECS problems with an EFS filesystem DNS name | AWS re:Post
unable to access the api from ecs fargate. Do we need to use different ip address for ecs fargate | AWS re:Post
Troubleshoot connection to AWS service endpoints from Linux | AWS re:Post
Hi guys, I was able to fix it by adding the dhcp option set with Google DNS server. I am not sure why it is not resolved by Amazon DNS. Let me know your comments if there is any complications with these approch. Thanks for all the answers.
We've encountered a similar issue - unable to resolve the www.couriersplease.com.au
host via the 10.0.0.2 default DNS resolver on an ECS Fargate instance. After hours of investigation, our only conclusion is that the ECS Fargate DNS resolver is broken in some way - perhaps it has negatively cached a missing record and is stuck.
Interestingly, we (very rarely) are able to resolve the domain via the default 10.0.0.2 resolver. But after a few seconds, it's broken again.
The domain is definitely resolvable from any other recursive resolver (eg, 1.1.1.1, 8.8.8.8) and dig +trace
shows a valid, sensible lookup (both from within the ECS Fargate instance, and on local dev machines).
Another interesting note is that the default DNS resolver for our VPC (accessed via 10.0.0.2 on an EC2 instance within our VPC) works fine. So it's just a problem with the resolver for ECS Fargate. Or perhaps some of the resolvers within a cluster, to explain the intermittent failures.
Relevant content
- asked 2 years ago
nslookup webpay3gint.transbank.cl
Server: 200.30.192.14 Address: 200.30.192.14#53
Non-authoritative answer: webpay3gint.transbank.cl canonical name = webpay3gint.tbk.akadns.net. Name: webpay3gint.tbk.akadns.net Address: 200.10.12.94