How do I troubleshoot connection issues between my Fargate task and other AWS services?

Short description

Applications that run inside a Fargate task with Amazon Elastic Container Service (Amazon ECS) can fail to access other AWS services for the following reasons:

• Insufficient AWS Identity and Access Management (IAM) permissions
• Incorrect subnet routes
• Network access control list (network ACL) restrictions
• Incorrect security group rules
• Incorrect Amazon Virtual Private Cloud (Amazon VPC) endpoint configurations

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Set up and use ECS Exec

Use ECS Exec to interact with your Fargate task's application container. Meet the prerequisites to use ECS Exec. Then, turn on ECS Exec for your Fargate task.

If your application container doesn't support shell sessions to run ECS Exec, then use an aws-cli Docker image to create a new Fargate task. The new task must be in the same subnet and VPC as your current Fargate task.

Configure connections between Amazon ECS and other AWS service endpoints

If your Amazon ECS tasks fail to connect to other services, then verify that you meet the following conditions:

• The endpoint URL is correct, including the protocol (http:// or https://), hostname, and port. Configure your VPC endpoints with subnets that align with your Amazon ECS service subnet configuration.
• The subnets that your Amazon ECS service uses must have the correct DNS configurations to resolve your endpoint URLs, either an Amazon provided DNS or custom resolver.
• The security group that's attached to your Amazon ECS tasks must allow outbound access to the destination service's port and protocol.
• Network access control lists (ACLs) in the subnet must allow traffic to and from the required service endpoints.

Test the connection between your Fargate task and AWS service endpoint

To test your Fargate task's connection to your AWS service endpoint and port, run the following telnet command from your container:

telnet endpoint-url port

Note: In the preceding command, replace endpoint-url with endpoint URL for your service and port with the port number for the service.

If the connection is successful, then you get the following example output:

Trying 10.0.1.169...  
Connected to sns.us-east-1.amazonaws.com.  
Escape character is '^]'.

To test the DNS resolution for your endpoint, run the following nslookup command:

nslookup example-endpoint

Example output:

Name: sns.us-east-1.amazonaws.com  
Address: 10.0.1.169

If you didn't install the telnet or nslookup commands in your container, then run one of the following commands to install them.

Debian-based containers:

apt-get update

apt-get install -y dnsutils telnet

Amazon Linux containers:

yum install -y bind-utils telnet

Troubleshoot connection timeout errors

If you receive connection timeout errors, then check your network configuration.

Security group rules

If traffic routes through a VPC endpoint, then the endpoint's security group must allow TCP traffic on port 443 from one of the following:

• The Amazon ECS task's security group
• The VPC CIDR block

Configure the task's security group to allow outbound traffic to the service's VPC endpoint.

VPC Routing

For public subnets, the Fargate task must have a default route of 0.0.0.0/0 that points to the internet gateway.

For private subnets, the route table must have a route to either a NAT gateway or a transit gateway according to your architecture.

Proxy settings

If your Amazon ECS task uses a proxy server, then make sure that the proxy configuration allows traffic to your service endpoint.

Analyze VPC flow logs

If you still experience connection issues, then use Amazon VPC Flow Logs to view the traffic flow details.

Troubleshoot IAM permissions issues

Check whether the Amazon ECS task role grants permissions to the application container to interact with other AWS services. 

For example, you might get the following errors when you try to connect to an Amazon Simple Notification Service (Amazon SNS) endpoint.

Example VPC endpoint policy error message:

"An error occurred (AuthorizationError) when calling the ListTopics operation: User: arn:aws:sts::123456789012:assumed-role/TaskRole/123456789012 is not authorized to perform: SNS:ListTopics on resource: arn:aws:sns:region-name:123456789012:* with an explicit deny in a VPC endpoint policy"

To resolve this issue, grant the Amazon VPC endpoint permission to perform the necessary actions against your AWS services. In the preceding example error message, the Amazon VPC endpoint must have the ListTopics permission.

Example task role permission error message:

"An error occurred (AuthorizationError) when calling the ListTopics operation: User: arn:aws:sts::123456789012:assumed-role/TaskRole/123456789012 is not authorized to perform: SNS:ListTopics on resource: arn:aws:sns:region-name:123456789012:* because no identity-based policy allows the SNS:ListTopics action"

To resolve this issue, grant the Amazon ECS task's IAM role has permissions to perform the necessary actions against the recipient services. In the preceding example error message, the IAM role must have the ListTopics permission.