How do I troubleshoot issues when I pass environment variables to my Amazon ECS task?

Short description

When you pass an environment variable or sensitive data inside your task, you might receive one of the following errors based on your configuration.

AWS Secrets Manager

"ResourceInitializationError error" or "AccessDenied error on Amazon Elastic Compute Cloud (Amazon EC2)".

To resolve these issues, see How do I troubleshoot issues related to AWS Secrets Manager secrets in Amazon ECS?

Amazon Elastic Block Store (Amazon EBS) volumes or sidecars

"ECS was unable to assume the configured ECS Infrastructure Role 'arn:aws:iam::111122223333:role/ecsInfrastructureRole'. Please verify that the role being passed has the proper trust relationship with Amazon ECS" or "ECS timed out while configuring the EBS volume attachment to your Task."

To resolve these issues, see Status reasons for Amazon EBS volume attachment to Amazon ECS tasks.

Amazon Simple Storage Service (Amazon S3)

"ResourceInitializationError: failed to download env files: file download command: non empty error stream".

To resolve this issue, complete the following resolution steps.

Parameter Store, a capability of AWS Systems Manager

"Fetching secret data from SSM Parameter Store in region: AccessDeniedException: User: arn:aws:sts::123456789:assumed-role/ecsExecutionRole/f512996041234a63ac354214 is not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:ap-south-1:12345678:parameter/status code: 400, request id: e46b40ee-0a38-46da-aedd-05f23a41e861". Or, "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secrets from ssm: service call has been retried 5 time(s): RequestCanceled".

To resolve these issues, complete the following resolution steps.

Note: It's a security best practice to store sensitive data in Secrets Manager secrets or Parameter Store parameters.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Determine the cause of the issue

Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook in the same AWS Region where your Amazon ECS cluster resources are located. Also, use the most recently failed task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation. By default, stopped ECS tasks are visible for 1 hour after they enter the Stopped state.

Based on the output of the automation, use one of the following manual troubleshooting steps.

Your Amazon ECS task execution role doesn't have the required IAM permissions

If you're using environment variables inside Parameter Store or Secrets Manager, then review AWS CloudTrail events for the following API calls:

• GetParameters for Parameter Store
• GetSecretValue for Secrets Manager

If you get the AccessDenied error for the task execution role, then you must manually add the required permissions as an inline policy to the role. You can also create a custom managed policy that you add to your Amazon ECS task execution role.

If you use an S3 bucket to store the environment variable as a .env file, then add the required permissions for Amazon S3 to the task execution role.

There are issues with your network configuration

If your task is in a private subnet, then verify the following configurations:

• The security group for the task or service allows outbound traffic on port 443.
• If you use a NAT gateway, then your task must have a default route to the NAT gateway.
• Your tasks use the required virtual private cloud (VPC) endpoints for Secrets Manager, Parameter Store, or Amazon S3.

Use the telnet command to verify connectivity to the Systems Manager, Secrets Manager, or the Amazon S3 endpoint.

Also, verify the following VPC endpoint configurations:

• The security group for your VPC endpoint allows outbound traffic from the task or service on port 443.
• The network access control list (network ACL) allows outbound traffic on port 443.
• You associated the VPC endpoint with the correct VPC.
• You activated the DNS hostnames and DNS resolution VPC attributes.

If your Amazon ECS task is in a public subnet, then verify the following configurations:

• You activated a public IP address for the task.
• The security group of your VPC allows outbound access on port 443 to the internet.
• The network ACL allows all inbound and outbound traffic between the subnets and the internet.

Your application can't read the environment variable

To check whether the correct environment variables populate in your task container, use ECS Exec to list all environment variables in your container. Activate ECS Exec, and then run the following command to interact with your container:

aws ecs execute-command --cluster cluster-name \
    --task task-id \
    --container container-name \
    --interactive \
    --command "/bin/sh"  

Note: Replace cluster-name with your cluster name, task-id with your task ID, and container-name with your container name.

Example output:

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-hlei32fctyur2bn63rhe2uraaq
sh-5.2#

Then, run the env command to list all the environment variables:

sh-5.2# env

Example output:

sh-5.2# env
AWS_EXECUTION_ENV=AWS_ECS_FARGATE
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

If you use the Amazon EC2 launch type, then you can also use the Docker exec command to interact with your container. To use Docker exec, complete the following steps:

1. Connect to the container instance where your task is running.

2. Run the following Docker command to find the container ID:

   docker container ps

3. To open the container, run the following Docker command:

   docker exec -it example-container-id bash

   Note: Replace example-container-id with your container ID. Also, select the shell based on your container's default shell.

4. Run the env command on your container to list all environment variables.

Make sure that environment variables that you defined in the task definition or .env file are in the list of variables.

The format of variable in the container definition is incorrect

When you define environment variables in the container definition, you must define the environment variables as key-value pair objects:

"environment": [
    {
        "name": "variable",    
        "value": "value"
    }
]

You must also use this format when you define the environment variables in your .env files.

The task doesn't automatically refresh the environment variable

The task doesn't automatically refresh environment variables in your running container that you update in your .env file or secret.

To add the updated environmental variables to your container instance, run the following update-service command:

aws ecs update-service --cluster example-cluster --service example-service --force-new-deployment

Note: Replace example-cluster with your cluster name and example-service with your service name.

If you use environment variables in your container definition, then you must create a new task definition to update the environment variables. Then, run the following update-service command to use the new task definition to create a new task or update your Amazon ECS service:

aws ecs update-service --cluster example-cluster --service example-service --task-definition family:revision

Note: Replace example-cluster with your cluster name, example-service with your service name, and family:revision with your new task definition.

When you pass environment variables to your task, Amazon ECS uses the following configurations:

• Environment variables that use the environment parameter in a container definition take precedence over the variables in an environment file.
• If you specify multiple environment files and they use the same variable, then the files process in the order of entry. Amazon ECS uses the first value of the variable, and ignores subsequent values of duplicate variables. It's a best practice to use unique variable names.
• If you specify an environment file as a container override, then Amazon ECS uses that file and ignores other environment files in the container definition.
• The environment variables are available to the PID 1 processes in a container from the /proc/1/environ file. If the container runs multiple processes or init processes, such as wrapper script or supervisord, then the environment variable is unavailable to non-PID 1 processes.

Related information

Pass sensitive data to an Amazon ECS container