How do I secure my Amazon ECS container instance with best practices and hardening techniques?

Short description

If you use the Amazon EC2 launch type in Amazon ECS, then EC2 instance issues such as terminations, reboots, or performance bottlenecks affect the container. To resolve these issues, implement best practices to secure your Amazon ECS container instances and address Amazon EC2 maintenance, scaling, and troubleshooting.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Secure your Amazon ECS container instances

Limit AWS account root user access

Don't use the root account unless required, for example to change account settings or modify the root user password. Instead, use AWS Identity and Access Management (IAM) roles and policies to set up access that adheres to the principal of least privilege. Also, set up multi-factor authentication (MFA) for the root account.

Implement security hardening

Use security groups to restrict inbound and outbound traffic to only the required IP addresses and ports. Regularly patch and update your Amazon ECS container instances. You can also use Amazon Inspector to identify vulnerabilities.

Use encryption for sensitive data in transit, such as HTTPS, and at rest, such as Amazon Elastic Block Storage (Amazon EBS) volumes. For more information, see Infrastructure protection.

Use AWS Systems Manager to streamline the patching and maintenance of Amazon ECS container instances. Use the preinstalled AWS Systems Manager Agent (SSM Agent) on Amazon ECS optimized Amazon Machine Images (AMIs) to implement automated patch management workflows. This includes scheduled maintenance windows and instance replacement strategies. For more information, see Automate patching by replacing Amazon ECS container instances.

Turn on logging and monitoring

Configure AWS CloudTrail to log API activities. It's a best practice to set up Amazon CloudWatch alarms to alert you of high CpuUtilization, DiskReadOps, DiskWriteOps, NetworkIn or NetworkOut values.

To monitor your container instance performance and resource usage, activate Container Insights.

Set up a backup and restore strategy

Use Amazon EBS snapshots or create AMIs as part of a disaster recovery plan. For more information, see Amazon EC2 backup and restore using AWS Backup.

Important: If you delete an EBS volume, then you can recover the volume only if you created an Amazon EBS snapshot or AMI backup.

To collect and export metadata, instance logs, and application data, use SSH to connect to your instance. Then, run the following describe-instances AWS CLI command to export the instance metadata:

aws ec2 describe-instances \
--instance-ids i-abcdxyz\
-- query "Reservations[].Instances[]" \
-- output text > file.txt

To export the instance logs, run the following get-console-output command:

aws ec2 get-console-output --instance-id i-abcdxyz --output text

Note: In the preceding commands, replace i-abcdxyz with your instance ID.

Troubleshoot EC2 instance issues

Troubleshoot unexpected terminations and reboots

Use Amazon ECS managed instance draining to make sure that your workloads gracefully stop and transition to stable instances during Spot interruptions or similar events.

For security and performance, Amazon EC2 regularly schedules routine maintenance events for your instances, such as reboots, stop and starts, and retirement. You can use the AWS Health Dashboard to track these events that can take several minutes to complete. For information about what actions you can take after routine maintenance events, see Recommended actions for Amazon EC2 instances affected by scheduled events.

If you experience a Spot Instance termination, then see Why did Amazon EC2 interrupt my Spot Instance?

Amazon EC2 might terminate your instance as part of scheduled maintenance. To manage this kind of instance termination, it's a best practice to use Amazon EC2 Auto Scaling lifecycle hooks.

Troubleshoot high resource utilization

To check for high CPU or memory usage, review the application logs and CpuUtilization, NetworkIn, NetworkOut, DiskReadOps, and DiskWriteOps CloudWatch metrics for your instance. Also, check the MemoryUtilized Container Insights metric.

If your configuration doesn't meet the resource needs of your instances, then scale up your instances.

Or, use Amazon ECS capacity providers to dynamically scale and optimize your cluster's compute resources. To create a capacity provider, use the Amazon ECS console. Or, run a create-capacity-provider command similar to the following example:

aws ecs create-capacity-provider --name "example-capacity=provider" --auto-scaling-group-provider "autoScalingGroupArn=arn:aws:autoscaling:us-east-1:123456789012:autoScalingGroup:a1b2c3d4-5678-90ab-cdef-EXAMPLE11111:autoScalingGroupName/MyAutoScalingGroup,managedScaling={status=ENABLED,targetCapacity=100,minimumScalingStepSize=1,maximumScalingStepSize=100},managedTerminationProtection=ENABLED"

Resolve insufficient disk space issues

The primary storage of the EC2 instance that's running Amazon ECS tasks can become full for the following reasons:

• Application logs
• Excessive container image storage
• Temporary files created by running container instances

Note that each file or directory uses one inode regardless of its size. As a result, if you have multiple small files, then you might exceed your inode quota. If you exceed your inode quota, then you can't create new files even when there's disk space available.

Use Amazon ECS configurations to automate task and image cleanup and remove stopped tasks and unused Docker images. Also, use the ECS_RESERVED_MEMORY variable to make sure that tasks don't consume all available memory.

To automatically clean up old or unused images in your repositories, use Amazon Elastic Container Registry (Amazon ECR) lifecycle policies. To clean up unused objects such as images, container instances, volumes, and networks, use Docker prune commands. For more information, see Prune unused Docker objects on the Docker Docs website.

To view the disk space that all mounted file systems use, run the following command:

df -h

To view the disk space that a specific file system uses, run the following command:

df -h /

Note: The preceding command shows data for only the root file system.

Resource issues might occur because you have few available inodes. To check your inode use, run the following command:

df -i

To view the total disk space that a specific directory uses, run the following command:

du -sh /path/to/directory

Note: Replace directory with your directory name.

To list the largest directories or files in your application, run the following command:

du -ah /path/to/directory | sort -rh | head -n 10

Note: The preceding command lists the top 10 largest files or directories. Replace 10 with the number of large files and directories that you want to view, and directory with your directory name.

To check the size of each subdirectory in your directory, run the following command:

du -h --max-depth=1

To optimize your available capacity, use Spot Fleet or capacity providers for your EC2 instances. For more information, see Powering your Amazon ECS clusters with Spot Fleet.

Related information

Fault tolerance and fault isolation

How can I troubleshoot connecting to my Amazon EC2 Linux instance using SSH?

Amazon EBS encryption

Best practices for Amazon EC2

Amazon ECS best practices