Questions tagged with Amazon Elastic Container Service

I have a container running in an EC2 instance on ECS. The container is hosting a django based application that utilizes S3 and RDS for its file storage and db needs respectively. I have appropriately configured my VPC, Subnets, VPC endpoints, Internet Gateway, roles, security groups, and other parameters such that I am able to host the site, connect to the RDS instance, and I can even access the site. The issue is with the connection to S3. When I try to run the command `python manage.py collectstatic --no-input` which should upload/update any new/modified files to S3 as part of the application set up the program hangs and will not continue. No files are transferred to the already set up S3 bucket. **Details of the set up:** All of the below is hosted on AWS Gov Cloud **VPC and Subnets** * 1 VPC located in Gov Cloud East with 2 availability zones (AZ) and one private and public subnet in each AZ (4 total subnets) * The 3 default routing tables (1 for each private subnet, and 1 for the two public subnets together) * DNS hostnames and DNS resolution are both enabled **VPC Endpoints** All endpoints have the "vpce-sg" security group attached and are associated to the above vpc * s3 gateway endpoint (set up to use the two private subnet routing tables) * ecr-api interface endpoint * ecr-dkr interface endpoint * ecs-agetn interface endpoint * ecs interface endpoint * ecs-telemetry interface endpoint * logs interface endpoint * rds interface endpoint **Security Groups** * Elastic Load Balancer Security Group (elb-sg) * Used for the elastic load balancer * Only allows inbound traffic from my local IP * No outbound restrictions * ECS Security Group (ecs-sg) * Used for the EC2 instance in ECS * Allows all traffic from the elb-sg * Allows http:80, https:443 from vpce-sg for s3 * Allows postgresql:5432 from vpce-sg for rds * No outbound restrictions * VPC Endpoints Security Group (vpce-sg) * Used for all vpc endpoints * Allows http:80, https:443 from ecs-sg for s3 * Allows postgresql:5432 from ecs-sg for rds * No outbound restrictions **Elastic Load Balancer** * Set up to use an Amazon Certificate https connection with a domain managed by GoDaddy since Gov Cloud route53 does not allow public hosted zones * Listener on http permanently redirects to https **Roles** * ecsInstanceRole (Used for the EC2 instance on ECS) * Attached policies: AmazonS3FullAccess, AmazonEC2ContainerServiceforEC2Role, AmazonRDSFullAccess * Trust relationships: ec2.amazonaws.com * ecsTaskExecutionRole (Used for executionRole in task definition) * Attached policies: AmazonECSTaskExecutionRolePolicy * Trust relationships: ec2.amazonaws.com, ecs-tasks.amazonaws.com * ecsRunTaskRole (Used for taskRole in task definition) * Attached policies: AmazonS3FullAccess, CloudWatchLogsFullAccess, AmazonRDSFullAccess * Trust relationships: ec2.amazonaws.com, ecs-tasks.amazonaws.com **S3 Bucket** * Standard bucket set up in the same Gov Cloud region as everything else **Trouble Shooting** If I bypass the connection to s3 the application successfully launches and I can connect to the website, but since static files are supposed to be hosted on s3 there is less formatting and images are missing. Using a bastion instance I was able to ssh into the EC2 instance running the container and successfully test my connection to s3 from there using `aws s3 ls s3://BUCKET_NAME` If I connect to a shell within the application container itself and I try to connect to the bucket using... ``` s3 = boto3.resource('s3') bucket = s3.Bucket(BUCKET_NAME) s3.meta.client.head_bucket(Bucket=bucket.name) ``` I receive a timeout error... ``` File "/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 179, in _new_conn raise ConnectTimeoutError( urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7f3da4467190>, 'Connection to BUCKET_NAME.s3.amazonaws.com timed out. (connect timeout=60)') ... File "/.venv/lib/python3.9/site-packages/botocore/httpsession.py", line 418, in send raise ConnectTimeoutError(endpoint_url=request.url, error=e) botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://BUCKET_NAME.s3.amazonaws.com/" ``` Based on [this article ](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html#vpc-endpoints-policies-s3) I think this may have something to do with the fact that I am using the GoDaddy DNS servers which may be preventing proper URL resolution for S3. > If you're using the Amazon DNS servers, you must enable both DNS hostnames and DNS resolution for your VPC. If you're using your own DNS server, ensure that requests to Amazon S3 resolve correctly to the IP addresses maintained by AWS. I am unsure of how to ensure that requests to Amazon S3 resolve correctly to the IP address maintained by AWS. Perhaps I need to set up another private DNS on route53? I have tried a very similar set up for this application in AWS non-Gov Cloud using route53 public DNS instead of GoDaddy and there is no issue connecting to S3. Please let me know if there is any other information I can provide to help.

In local, I was able to connect to the container through the docker exec command. However, fargate is a serverless service, so I wonder if it can connect to the container.

When using Step Functions with a container service like EKS or ECS how does output work for those steps? From the below two linked questions it seems as if input is done by either passing the input to the command override or by passing it as an environment variable. How do these jobs return output to be consumed by the following step? Input questions: * https://repost.aws/questions/QUQqIMl4s6QFKWtGeABPhgVA/step-functions-pass-input-into-fargate-instance * https://repost.aws/questions/QUCWMj_HPIQi6nmpUrbS3UTQ/step-functions-lambda-and-ecs **EDIT** According to this blog there isn't a way to output from ECS or Fargate tasks. So the outputs need to be deterministic based on the previous inputs. Can anyone confirm this and does this go for EKS as well? https://nuvalence.io/blog/aws-step-function-integration-with-ecs-or-fargate-tasks-data-in-and-out

Hello, We encountered troubles with ECS on Docker on EC2. Some tasks stay a long time in provisionning state while an EC2 is available and others tasks with the same docker image and parameters succeed to start. Then, the status goes from PROVISIONNING to STOPPED with reason : "Task failed to start". I have no logs, or info on the containers in ECS console and Cloudwatch, and did not find any tips in the EC2 logs... Any idea on why we have this behaviour ? or a way to have more traces to investigate ? Thanks Valentin

I have a Lambda function that is supposed to pass its own permissions to the code running in an ECS task. It looks like this: ``` ecs_parameters = { "cluster": ..., "launchType": "FARGATE", "networkConfiguration": ..., "overrides": { "taskRoleArn": boto3.client("sts").get_caller_identity().get("Arn"), ... }, "platformVersion": "LATEST", "taskDefinition": f"my-task-definition-{STAGE}", } response = ecs.run_task(**ecs_parameters) ``` When I run this in Lambda, i get this error: ``` "errorMessage": "An error occurred (ClientException) when calling the RunTask operation: ECS was unable to assume the role 'arn:aws:sts::787364832896:assumed-role/my-lambda-role...' that was provided for this task. Please verify that the role being passed has the proper trust relationship and permissions and that your IAM user has permissions to pass this role." ``` If I change the task definition in ECS to use `my-lambda-role` as the task role, it works. It's specifically when I try to override the task role from Lambda that it breaks. The Lambda role has the `AWSLambdaBasicExecutionRole` policy and also an inline policy that grants it `ecs:runTask` and `iam:PassRole`. It has a trust relationship that looks like: ``` "Effect": "Allow", "Principal": { "Service": [ "ecs.amazonaws.com", "lambda.amazonaws.com", "ecs-tasks.amazonaws.com" ] }, "Action": "sts:AssumeRole" ``` The task definition has a policy that grants it `sts:AssumeRole` and `iam:PassRole`, and a trust relationship that looks like: ``` "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com", "AWS": "arn:aws:iam::account-ID:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS" }, "Action": "sts:AssumeRole" ``` How do I allow the Lambda function to pass the role to ECS, and ECS to assume the role it's been given? P.S. - I know a lot of these permissions are overkill, so let me know if there are any I can get rid of :) Thanks!

Hello, i have some issue on SES, i send email with SES API v2 using nodejs, but i'm send file more than 10mb and then i'm got error InvalidParameterValue: Message length is more than 10485760 bytes. i'm reading what new aws https://aws.amazon.com/about-aws/whats-new/2022/04/amazon-ses-v2-supports-email-size-40mb-inbound-outbound-emails-default/ does anyone know this problem?

Hi everyone, I just wanted to get feedback on my proposed solution for a multi-region ECS dockerized app. Currently we have the following resources in Region A: ``` Postgres DB (Used for user accounts only) Backend+Frontend NextJS App (Dockerized) ECS Backend Microservice App for conversion of files (Dockerized) ECS Backend 3rd party API + Datastore (This resource is also deployed in other regions) Unknown architecture ``` I now need to deploy to Regions B and C. The Backend 3rd party API is already deployed in these regions. I am thinking of deploying the following resources to the following regions: ``` Backend+Frontend NextJS App (Dockerized) Backend Microservice App for conversion of files (Dockerized) ``` Our app logs in the user (authentication + authorization) using the 3rd party API, and after login we can see which region their data is in. So after login I can bounce them + their token to the appropriate region. I cannot use Route53 routing reliably because the Source of Truth about their region is available after login, and, for example, they may be (rarely) accessing from region B (if they are travelling) while their datastore is in region C (In which case I need to bounce them to region C). I also don't need to replicate our database to other regions because it only stores their account information for billing purposes, so the performance impact is minimal and only checked on login/logout. Currently we have low 10s of users, so I can easily restructure and deploy a different architecture if/when we start scaling. Critique is welcome!

Overview: I'm trying to run an XGboost model on a bunch of parquet files sitting in S3 using dask by setting up a fargate cluster and connecting it to a Dask cluster. Total dataframe size totals to about 140 GB of data. I scaled up a fargate cluster with properties: Workers: 40 Total threads: 160 Total memory: 1 TB So there should be enough data to hold the data tasks. Each worker has 9+ GB with 4 Threads. I do some very basic preprocessing and then I create a DaskDMatrix which does cause the task bytes per worker to get a little high, but never above the threshold where it would fail. Next I run xgb.dask.train which utilizes the xgboost package not the dask_ml.xgboost package. Very quickly, the workers die and I get the error `XGBoostError: rabit/internal/utils.h:90: Allreduce failed`. When I attempted this with a single file with only 17MB of data, I would still get this error but only a couple workers die. Does anyone know why this happens since I have double the memory of the dataframe? ``` X_train = X_train.to_dask_array() X_test = X_test.to_dask_array() y_train = y_train y_test = y_test ``` dtrain = xgb.dask.DaskDMatrix(client,X_train, y_train) output = xgb.dask.train( client, {"verbosity": 1, "tree_method": "hist", "objective": "reg:squarederror"}, dtrain, num_boost_round=100, evals=[(dtrain, "train")])`

I have an image I deploy to ECS that expects an environment variable called `DATABASE_URL` which contains the username and password as the userinfo part of the url (e.g. `postgres://myusername:mypassword@mydb.foo.us-east-1.rds.amazonaws.com:5432/mydbname`). I cannot change the image. Using `DatabaseInstance.Builder.credentials(fromGeneratedSecret("myusername"))`, CDK creates a secret in Secrets Manager for me that has all of this information, but not as a single value: ```json { "username":"myusername", "password":"mypassword", "engine":"postgres", "host":"mydb.foo.us-east-1.rds.amazonaws.com", "port":5432, "dbInstanceIdentifier":"live-myproduct-db" } ``` Somehow I need to synthesise that `DATABASE_URL` environment variable. I don't think I can do it in the ECS Task Definition - as far as I can tell the secret can only reference a single key in a secret. I thought I might be able to add an extra `url` key to the existing secret using references in cloud formation - but I can't see how. Something like: ```java secret.newBuilder() .addTemplatedKey( "url", "postgres://#{username}:#{password}@#{host}:#{port}/#{db}" ) .build() ``` except that I just made that up... Alternatively I could use CDK to generate a new secret in either Secrets Manager or Systems Manager - but again I want to specify it as a template so that the real secret values don't get materialised in the CloudFormation template. Any thoughts? I'm hoping I'm just missing some way to use the API to build compound secrets...

I'm trying to use a boto3 ECS waiter to wait on a Fargate ECS task to complete. The vast majority of the time the waiter works as expected (waits for the task to reach the STOPPED status). However, sporadically the waiter will return a failure because a task is marked as missing. However, I can find the task itself in the cluster and cloudwatch logs for the task. When I first encountered this, I switched to using boto3 [`ecs.describe_tasks`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.describe_tasks) method to see if I could get more information about what was happening. When the above situation occurs, descirbe_tasks returns something like: ``` {'tasks': [], 'failures': [\{'arn': 'arn:aws:ecs:us-west-2:21234567891011:task/something-something/dsfsadfasdhfasjklhdfkdsajhf', 'reason': 'MISSING'}], 'ResponseMetadata': \{'RequestId': sdkfjaskdjfhaskdjfhasd', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'sdkjfhsdkajhfksadhkjsadf', 'content-type': 'application/x-amz-json-1.1', 'content-length': '145', 'date': 'Fri, 06 May 2022 08:36:11 GMT'}, 'RetryAttempts': 0}} ``` I've looked at the [AWS Docs ](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/api_failures_messages.html) and the none of the scenarios outlined for `reason: MISSING` apply in my circumstance. I'm passing my cluster name as an argument to the call as well. Since this happens intermittently its difficult to troubleshoot. What does the MISSING status mean? What are the reasons why an API call to check on task status would return missing when the task exists?

Hi, As mentioned in the question, my ECS tasks cannot connect to my RDS. The ECS tasks try to resolve the rds by name, and it resolves to the RDS public IP (RDS has public and private IPs). However, the security group on RDS doesn't allow open access from all IPs so the connection fails. I temporarily allowed all connections and could see that the ECS tasks are routing through the open internet to access the RDS. Reachability Analyzer checking specific tasks' Elastic Network Interface to the RDI ENI is successful, using internal routing through the peering connection. At the same time I have another server on VPC C that can connect to the RDS. All the config is similar between these two apps, including the peering connection, security group policies and routing tables. Any help is appreciated Here are some details about the VPCs VPC A - 15.2.0.0/16 [three subnets] VPC B - 111.30.0.0/16 [three subnets] VPC C - 15.0.0.0/16 [three subnets] Peering Connection 1 between A and B Peering Connection 2 between C and B Route table for VPC A: 111.30.0.0/16 : Peering Connection 1 15.2.0.0/16: Local 0.0.0.0/0: Internet Gateway Route table for VPC C: 111.30.0.0/16: Peering Connection 2 15.2.0.0/16: Local 0.0.0.0/0: Internet Gateway Security groups allow traffic to RDS: Ingress: 15.0.0.0/16: Allow DB Port 15.2.0.0/16: Allow DB Port Egress: 0.0.0.0/0: Allow all ports When I add the rule: 0.0.0.0/0 Allow DB Port to the RDS, then ECS can connect to my RDS through its public IP.

I have an ECS service that exposes port 8080. I want to have the load balancer, target groups and target use that port as opposed to port 80. Here is a snippet of my code: ``` const servicePort = 8888; const metricsPort = 8888; const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef'); const repository = ecr.Repository.fromRepositoryName(this, 'cloud-config-server', 'cloud-config-server'); taskDefinition.addContainer('Config', { image: ecs.ContainerImage.fromEcrRepository(repository), portMappings: [{containerPort : servicePort, hostPort: servicePort}], }); const albFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'AlbConfigService', { cluster, publicLoadBalancer : false, taskDefinition: taskDefinition, desiredCount: 1, }); const applicationTargetGroup = new elbv2.ApplicationTargetGroup(this, 'AlbConfigServiceTargetGroup', { targetType: elbv2.TargetType.IP, protocol: elbv2.ApplicationProtocol.HTTP, port: servicePort, vpc, healthCheck: {path: "/CloudConfigServer/actuator/env/profile", port: String(servicePort)} }); const addApplicationTargetGroupsProps: elbv2.AddApplicationTargetGroupsProps = { targetGroups: [applicationTargetGroup], }; albFargateService.loadBalancer.addListener('alb-listener', { protocol: elbv2.ApplicationProtocol.HTTP, port: servicePort, defaultTargetGroups: [applicationTargetGroup]} ); } } ``` This does not work. The health check is taking place on port 80 with the default URL of "/" which fails, and the tasks are constantly recycled. A target group on port 8080, with the appropriate health check, is added, but it has no targets. What is the recommended way to achieve load balancing on a port other than 80? thanks

Scenario: * Using ECS Fargate to launch Tasks from a Task Definition ( the container is a Python Django container, running a management command if this matters) * Problem: Logs were empty in Cloudwatch for the task run (completely - no data at all, but the log groups and stream were created). * When running the container locally, stdout outputted the expected data After much research and trial and error, I found that using a ENTRYPOINT command with a shell script is the problem. If I were to switch this to run the container with just the raw cli command as the ENTRYPOINT, the logs show in CW as expected. What is the reasoning/problem for this, and is there a different/better way to find out what was going on here besides trial and error? This Doesn't Work (in docker) ``` ENTRYPOINT [ "/app/docker-entrypoint.sh" ] CMD ["run_management_command"] ``` ``` # docker-entrypoint.sh # code shortened for a little brevity, may have syntax errors shift exec python manage.py "$@" ``` This Works ( in docker) ``` CMD ["python", "manage.py", "run_management_command"] ``` Does the shell script / exec cause the AWS std out logger to become disassociated somehow?

I'm quite new at AWS and use mostly the console to build my project. I have placed a containerized Streamlit web app in an AWS EC2/ECS instance beyond an ALB (https listener with session timeout 3960 secs.) and let users access it through Cognito authentication with Authorization code grant. Everything works fine, users are allowed to the app. Now, I would like users to be authomatically logged out after 60 minutes and redirected to the signout URL. I've set the refresh token expiration at 60 min., the access token and ID token expiration at 5 min. However, the backend continues delivering data to logged in users even after 60 minutes, so my idea doesn't work. Then, I've implemented a Lambda function with admin_user_global_sign_out but it doesn't work either: users do still get data from the backend. I'm wondering what I shall do and looking for a solution that I can implement using the AWS console so that the procedure is clear to me. Thank you for any help.

We recently migrated our service to ECS, and we’ve seen a pattern of errors like this a few times: 1. Our CPU usage is low, so as part of normal autoscaling, ECS starts to reduce the number of tasks by 1 2. ECS claims to have deregistered 1 target and be draining connections 3. 90 seconds later (after the deregistration delay) ECS stops the task 4. Immediately after the task is stopped, a flood of load balancer 502s happens, all directed at just one IP. We suspect that this is the IP of the task that was removed and stopped, but somehow not removed from the ELB target group We don’t have any long-lived connections, so the 90 second deregistration delay should be long enough for the task to finish processing its requests before it’s stopped. It seems that the task selected for removal isn’t actually removed from the ALB target group, even though the ECS logs include messages indicating that it is. The logs include ``` * service \[our-service\] deregistered 1 targets in target-group \[our-target-group\] followed by * service \[our-service\] has begun draining connections on 1 tasks. ``` Events like this happen rarely (definitely not every time we scale down) but frequently enough to notice. Does anybody have ideas about why these errors might be happening, or how to get more information about what is going on? Thanks in advance.

I am trying to implement a proxy to our Aurora instance, but having difficulty getting the IAM access to work properly. We have a microservice in an ECS container that is attempting to access the database. The steps I've followed so far: * Created a secret containing the DB credentials * Created the proxy with the following config options: * Engine compatibility: MySQL * Require TLS - enabled * Idle timeout: 20 minutes * Secret - Selected DB credential secret * IAM Role - Chose to create new role * IAM Authentication - Required * Modified the policy of the proxy IAM role as per the details on [this page](https://aws.amazon.com/getting-started/hands-on/set-up-shared-database-connection-amazon-rds-proxy/). * Enabled enhanced logging When issuing GET requests to the microservice, I see the following in the CloudWatch logs: > Credentials couldn't be retrieved. The IAM role "arn:our-proxy-role" is not authorized to read the AWS Secrets Manager secret with the ARN "arn:our-db-credential-secret" Another interesting wrinkle to all of this: I pulled up the policy simulator, selecting the RDS proxy role and all of the actions under the Secrets Manager service, and all actions show up as being allowed. I would sincerely appreciate any kind of guidance to indicate what I'm missing here.

I create a ECS service using EC2 instances, then i create an Application Load Balancer and a target group, my docker image the task definition its using follow configuration: ```json { "ipcMode": null, "executionRoleArn": null, "containerDefinitions": [ { "dnsSearchDomains": null, "environmentFiles": null, "logConfiguration": { "logDriver": "awslogs", "secretOptions": null, "options": { "awslogs-group": "/ecs/onestapp-task-prod", "awslogs-region": "us-east-2", "awslogs-stream-prefix": "ecs" } }, "entryPoint": null, "portMappings": [ { "hostPort": 0, "protocol": "tcp", "containerPort": 80 } ], "cpu": 0, "resourceRequirements": null, "ulimits": null, "dnsServers": null, "mountPoints": [], "workingDirectory": null, "secrets": null, "dockerSecurityOptions": null, "memory": null, "memoryReservation": 512, "volumesFrom": [], "stopTimeout": null, "image": "637960118793.dkr.ecr.us-east-2.amazonaws.com/onestapp-repository-prod:5ea9baa2a6165a91c97aee3c037b593f708b33e7", "startTimeout": null, "firelensConfiguration": null, "dependsOn": null, "disableNetworking": null, "interactive": null, "healthCheck": null, "essential": true, "links": null, "hostname": null, "extraHosts": null, "pseudoTerminal": null, "user": null, "readonlyRootFilesystem": false, "dockerLabels": null, "systemControls": null, "privileged": null, "name": "onestapp-container-prod" } ], "placementConstraints": [], "memory": "1024", "taskRoleArn": null, "compatibilities": [ "EXTERNAL", "EC2" ], "taskDefinitionArn": "arn:aws:ecs:us-east-2:637960118793:task-definition/onestapp-task-prod:25", "networkMode": null, "runtimePlatform": null, "cpu": "1024", "revision": 25, "status": "ACTIVE", "inferenceAccelerators": null, "proxyConfiguration": null, "volumes": [] } ``` The service its using ALB and its using same Target Group as ALB, my task its running, and i can access using public ip from instance, but the target group does not have registered my tasks.

Hello, I'd like to deploy SageMaker's built-in algorithm, BlazingText model on Fargate instead of Sagemaker endpoint. So, I tried to make an ECS task using BlazingText docker path. Here is my CDK code for it. const loadBalancedFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', { memoryLimitMiB: 1024, desiredCount: 1, cpu: 512, taskImageOptions: { image: ecs.ContainerImage.fromRegistry("811284229777.dkr.ecr.us-east-1.amazonaws.com/blazingtext:1"), }, }); However, I got an error: CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "811284229777.dkr.ecr.us-east-1.amazonaws.com/blazingtext:1": pulling from host 811284229777.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests 1]... Is it impossible to pull docker container of sagemaker built-in algorithm from ECS?

Is it possible for us to mount an AWS FSx for Windows File Server file share onto our ECS EC2 host? Today, we have an application hosted on a Linux server with those FSx file shares mounted directly. Now that we are attempting to containerize these applications, we've come across this potential roadblock. Is this possible with ECS on EC2? Or, better yet, is this possible with ECS on Fargate? Thank you!

I have updated several task definitions and for the life of me I cannot get the latest revisions to run. We are using the AWS code management system, e.g. Code Deploy, Code Commit, etc. Wondering if there is a preferred way to get task definition to run the latest revision. I have tried stopping the service and updating with "Force new Deployment" checked and so far no dice. Any idea what I'm doing wrong?

My application allows uploading files with several simultaneous requests. Requests arrive at the CDN, go to the load balancer, and the load balancer passes them on to the API docker container, which is on the Elastic Container Service (ECS). Some requests complete successfully, others give 504 and 502 errors, intermittently. When I change the CDN origin to go straight to the container, all requests work with fast response. So I think the problem is in the Application Load Balancer. I already increased the idle timeout in the ALB, but the problem remains the same. I also tried increasing the Origin Connection Timeout in CloudFront, but it didn't work either. I have only 1 Docker container in the service. Can someone help me on this issue? Best regards! Fernando

I have an ECS Fargate task running an ASP.NET image and I have noticed that it keeps failing due to OOM exception. I can look in CloudWatch and the memory keeps increasing. When I run in a local docker container, I can see that the garbage collector is running correctly and there is no memory problem. I have seen people suggest to set Server GC to false, but I'm not sure if that is the problem. From what I have read this should have been fixed in previous versions (I am using asp.net 5). Is this expected and is the correct approach to set this to false?

Hi there i am trying to start a task which uses gpu on my instance. EC2 is already added to a cluster but it failed to start, here is the error: ``` status: STOPPED (CannotStartContainerError: Error response from dae) Details Status reason CannotStartContainerError: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr Network bindings - not configured ``` ec2: setup ``` Type: AWS::EC2::Instance Properties: IamInstanceProfile: !Ref InstanceProfile ImageId: ami-0d5564ca7e0b414a9 InstanceType: g4dn.xlarge KeyName: tmp-key SubnetId: !Ref PrivateSubnetOne SecurityGroupIds: - !Ref ContainerSecurityGroup UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=traffic-data-cluster >> /etc/ecs/ecs.config echo ECS_ENABLED_GPU_SUPPORT=true >> /etc/ecs/ecs.config ``` Dockerfile ``` FROM nvidia/cuda:11.6.0-base-ubuntu20.04 ENV NVIDIA_VISIBLE_DEVICES all ENV NVIDIA_DRIVER_CAPABILITIES compute,utility # RUN nvidia-smi RUN echo 'install pip packages' RUN apt-get update RUN apt-get install python3.8 -y RUN apt-get install python3-pip -y RUN ln -s /usr/bin/python3 /usr/bin/python RUN pip3 --version RUN python --version WORKDIR / COPY deployment/video-blurring/requirements.txt /requirements.txt RUN pip3 install --upgrade pip RUN pip3 install --user -r /requirements.txt ## Set up the requisite environment variables that will be passed during the build stage ARG SERVER_ID ARG SERVERLESS_STAGE ARG SERVERLESS_REGION ENV SERVER_ID=$SERVER_ID ENV SERVERLESS_STAGE=$SERVERLESS_STAGE ENV SERVERLESS_REGION=$SERVERLESS_REGION COPY config/env-vars . ## Sets up the entry point for running the bashrc which contains environment variable and ## trigger the python task handler COPY script/*.sh / RUN ["chmod", "+x", "./initialise_task.sh"] ## Copy the code to /var/runtime - following the AWS lambda convention ## Use ADD to preserve the underlying directory structure ADD src /var/runtime/ ENTRYPOINT ./initialise_task.sh ```

I am researching best practices when implementing Capacity Providers for ECS Clusters. One of the AWS reference videos recommends the usage of one Auto-Scaling Group / Capacity Provider per AZ. What is the main benefit of this recommendation other than distributing tasks at the capacity provider level ? Are there any other best practices to be aware of when implementing Capacity Providers ?

I have an image that binds to port 80 as a **non-root** user. I can run it locally (macOS Monterey, Docker Desktop 4.7.1) absolutely fine. When I try and run it as part of an ECS service on Fargate it fails as so: **Failed to bind to 0.0.0.0/0.0.0.0:80** **caused by SocketException: Permission denied** Fargate means I have to run the task in network mode `awsvpc` - not sure if that's related? Any views on what I'm doing wrong? The [best practices document](https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/bestpracticesguide.pdf) suggests that I should be running as non-root (p.83) and that under awsvpc it's reasonable to expose port 80 (diagram on p.23). FWIW here's a mildly cut down version the JSON from my task definition: ``` { "taskDefinitionArn": "arn:aws:ecs:us-east-1:<ID>:task-definition/mything:2", "containerDefinitions": [ { "name": "mything", "image": "mything:latest", "cpu": 0, "memory": 1024, "portMappings": [ { "containerPort": 80, "hostPort": 80, "protocol": "tcp" } ], "essential": true, "environment": [] } ], "family": "mything", "executionRoleArn": "arn:aws:iam::<ID>:role/ecsTaskExecutionRole", "networkMode": "awsvpc", "revision": 2, "volumes": [], "status": "ACTIVE", "requiresAttributes": [ { "name": "com.amazonaws.ecs.capability.logging-driver.awslogs" }, { "name": "ecs.capability.execution-role-awslogs" }, { "name": "com.amazonaws.ecs.capability.ecr-auth" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19" }, { "name": "ecs.capability.execution-role-ecr-pull" }, { "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18" }, { "name": "ecs.capability.task-eni" } ], "placementConstraints": [], "compatibilities": [ "EC2", "FARGATE" ], "runtimePlatform": { "operatingSystemFamily": "LINUX" }, "requiresCompatibilities": [ "FARGATE" ], "cpu": "256", "memory": "1024", "tags": [] } ```

I have an ECS Fargate service with a security group associated. This security group is only used by the Fargate service. If I decided to delete the service and the security group I'm having an error deleting the security group saying that the security group has some dependant objects. I get that this is due to the network interface created by the ECS service. What I would like to understand is how long it takes to AWS/ECS to clean these network interfaces after the services are deleted? I've tried to add a waiter after I delete the service waiting for the network interface to be gone but it is too variable and sometimes it can take more than 12 min!! Do I really need to wait until this network interface is gone to delete the security group? Is there another resource I can check to know that I can delete the security group? Is there a way to predict/make faster the deletion of the internal network interfaces created by ECS? It is quite annoying not having control over this.

Hi there, I am trying to add an ec2 instance in to an ecs cluster, I tried to follow this guild https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html , but i could not make it working. here is my ec2 config: ``` EC2: Type: AWS::EC2::Instance Properties: ImageId: ami-0d5564ca7e0b414a9 InstanceType: g4dn.xlarge KeyName: tmp-key SubnetId: !Ref PrivateSubnetOne SecurityGroupIds: !Ref ContainerSecurityGroup UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=traffic-data-cluster >> /etc/ecs/ecs.config echo ECS_ENABLED_GPU_SUPPORT=true >> /etc/ecs/ecs.config ``` Thank for your help

I have a CloudFormation setup with Scheduled Actions to autoscale services based on times. There is one action that scales up to start the service, and another to scale down to turn it off. I also occasionally add an additional action to scale up if a service is needed at a different time on a particular day. I'm having an issue where my service is being scaled down instead of up when I specify this additional action. Looking at the console logs I get an event that looks like: ``` 16:00:00 -0400 Message: Successfully set min capacity to 0 and max capacity to 0 Cause: scheduled action name ScheduleScaling_action_1 was triggered ``` However the relevant part of the CloudFormation Template for the Scheduled Action with the name in the log has a different time, e.g.: ``` { "ScalableTargetAction": { "MaxCapacity": 0, "MinCapacity": 0 }, "Schedule": "cron(0 5 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_1" } ``` What is odd is that the time this action is triggering matches exactly with the Schedule time for another action. E.g. ``` { "ScalableTargetAction": { "MaxCapacity": 1, "MinCapacity": 1 }, "Schedule": "cron(00 20 ? * 2-5 *)", "ScheduledActionName": "ScheduleScaling_action_2" } ``` I am using CDK to generate the CloudFormation template, which doesn't appear to allow me to specify a timezone. So my understanding is that the times here should be UTC. What could cause the scheduled action to trigger at the incorrect time like this?

App Runner actions work very slow for me. create/pause/resume may take 2-5 minutes for simple demo image (`public.ecr.aws/aws-containers/hello-app-runner:latest`) and create-service when image not found takes ~10 minutes: example #1 - 5 min to deploy hello-app image ``` 04-17-2022 05:59:55 PM [AppRunner] Service status is set to RUNNING. 04-17-2022 05:59:55 PM [AppRunner] Deployment completed successfully. 04-17-2022 05:59:44 PM [AppRunner] Successfully routed incoming traffic to application. 04-17-2022 05:58:33 PM [AppRunner] Health check is successful. Routing traffic to application. 04-17-2022 05:57:01 PM [AppRunner] Performing health check on port '8000'. 04-17-2022 05:56:51 PM [AppRunner] Provisioning instances and deploying image. 04-17-2022 05:56:42 PM [AppRunner] Successfully pulled image from ECR. 04-17-2022 05:54:56 PM [AppRunner] Service status is set to OPERATION_IN_PROGRESS. 04-17-2022 05:54:55 PM [AppRunner] Deployment started. ``` example #2 - 10 min when image not found ``` 04-17-2022 05:35:41 PM [AppRunner] Failed to pull your application image. Be sure you configure your service with a valid access role to your ECR repository. 04-17-2022 05:25:47 PM [AppRunner] Starting to pull your application image. ``` example #3 - 10 min when image not found ``` 04-17-2022 06:46:24 PM [AppRunner] Failed to pull your application image. Be sure you configure your service with a valid access role to your ECR repository. 04-17-2022 06:36:31 PM [AppRunner] Starting to pull your application image. ``` but 404 error should be detected immediately and fail much faster. because no need to retry 404 many times for 10 min, right? additionally the error message `Failed to pull your application image. Be sure you configure your service with a valid access role to your ECR repository` is very confusing. it doesn't show image name and doesn't provide the actual cause. 404 is not related to access errors like 401 or 403, correct? can App Runner actions performance and error message be improved?

Hello, I'm trying to understand why an ECS cluster configured with a capacity provider for EC2 with no defined service or running tasks, an empty cluster then, has a CapacityProviderReservation at 100 at all times. Following the explanation of https://aws.amazon.com/fr/blogs/containers/deep-dive-on-amazon-ecs-cluster-auto-scaling/ CapacityProviderReservation = M / N x 100 where N = Already running instances and M = Needed instances In my case, there is no needed nor running instances because there is nothing to run so N and M are equal 0, so why is CapacityProviderReservation always at 100 ? When there is only one server in the autoscaling group there is no issue, but if the max capacity is > 1 then when I start a task CapacityProviderReservation = 200 so 2 instances are started when only is needed. If anyone can enlighten me I would be very grateful :) Valentin

I'm getting this error when I try to connect to my Fargate ECS container using ECS Exec feature: ``` An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later. ``` The task's role was missing the SSM permissions, I added them, but the error persists. Below is the output of the [amazon-ecs-exec-checker](https://github.com/aws-containers/amazon-ecs-exec-checker) tool: ``` Prerequisites for check-ecs-exec.sh v0.7 ------------------------------------------------------------- jq | OK (/usr/bin/jq) AWS CLI | OK (/usr/local/bin/aws) ------------------------------------------------------------- Prerequisites for the AWS CLI to use ECS Exec ------------------------------------------------------------- AWS CLI Version | OK (aws-cli/2.2.1 Python/3.8.8 Linux/5.13.0-39-generic exe/x86_64.ubuntu.20 prompt/off) Session Manager Plugin | OK (1.2.295.0) ------------------------------------------------------------- Checks on ECS task and other resources ------------------------------------------------------------- Region : us-west-2 Cluster: removed Task : removed ------------------------------------------------------------- Cluster Configuration | Audit Logging Not Configured Can I ExecuteCommand? | arn:aws:iam::removed ecs:ExecuteCommand: allowed ssm:StartSession denied?: allowed Task Status | RUNNING Launch Type | Fargate Platform Version | 1.4.0 Exec Enabled for Task | OK Container-Level Checks | ---------- Managed Agent Status ---------- 1. RUNNING for "api-staging" ---------- Init Process Enabled (api-staging:14) ---------- 1. Enabled - "api-staging" ---------- Read-Only Root Filesystem (api-staging:14) ---------- 1. Disabled - "api-staging" Task Role Permissions | arn:aws:iam::removed ssmmessages:CreateControlChannel: allowed ssmmessages:CreateDataChannel: allowed ssmmessages:OpenControlChannel: allowed ssmmessages:OpenDataChannel: allowed VPC Endpoints | Found existing endpoints for vpc-removed: - com.amazonaws.us-west-2.s3 SSM PrivateLink "com.amazonaws.us-west-2.ssmmessages" not found. You must ensure your task has proper outbound internet connectivity. Environment Variables | (api-staging:14) 1. container "api-staging" - AWS_ACCESS_KEY: not defined - AWS_ACCESS_KEY_ID: defined - AWS_SECRET_ACCESS_KEY: defined ``` The output has only green and yellow markers, no red ones, which means Exec should work. But it doesn't. Any ideas why?

We have * an ecs cluster * with a managed auto scaling group (aws_ecs_capacity_provider) * and a aws_launch_template to create EC2 instances we manually turned off the scale in protection in the advanced settiongs of the ASG. But every new EC2 machine that is created by the ASG is again scale-in-protected. Problem: we want to use scheduled actions to automatically set the desired capacity periodically up and down. And the reduction is not working when we do not manually switch off the scale-in-protection every time an the EC2 instances. This seems like a bug in AWS, because the property 'protect_from_scale_in' on the ASG has no effect! Please help! Kind regards,

We are mounting EFS volume to our task and we are running the task on fargate but it giving above mentioned error. we made sure everything running the Fargate task and EFS is in the same VPC, using the same sub net and availability zone. We also opened port 2049 for NFS. We also tried opening all traffic for all ports and attached to the task but still we are facing the same issue.

I created ecs and service, and in the event tab, I could see the following phrase. (fargate) "service xxxx was unable to place a task because your account is currently blocked." It's the first time I've seen this phrase, and I'm not sure how to deal with it.

I have set up a node app following the guide here https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html Both containers are running but the X-ray daemon didn't receive data from the nodejs app. I check the x-ray daemon logs and found the below logs. Can anyone help with what I missed please? ``` [Error] Get instance id metadata failed: RequestError: send request failed caused by: Get http://169.254.169.254/latest/meta-data/instance-id: dial tcp 169.254.169.254:80: connect: invalid argument ```

Hi, I am new to AWS Amplify and would like guidance on how to send a query to an ***existing*** GraphQL API on AWS AppSync. I am unsure how to start as a lot of Amplify coverage creates a *new* AppSync API using the Amplify CLI. ## Objectives * Set up a Node.js project to work with an existing AWS AppSync API, using AWS Amplify as the GraphQL client. * Send a single query to an existing AWS AppSync API. The query lists game results from a DynamoDB table and is called `listGames` in my GraphQL schema. * I need to repeat the query in order to fetch all available database records that satisfy the query. This would mean adding results to an array/object until the `nextToken` is `null` (i.e. no more records can be found for the query). ## Context * This application is deployed in an Amazon ECS container using AWS Fargate. * The ECS service is fronted by an Application Load Balancer (ALB). * A leader board web page fetches game results through a `POST` request to the ALB's DNS name / URL and adds them to a HTML table. ## Notes * For now, API key is my authentication method. I would soon like to switch to a task IAM role in ECS. * The ECS deployment described in 'Context' is working but it sends `POST` requests without AWS libraries. It is my understanding that I would need to use an AWS library in order to use an IAM role for AppSync authentication (used as a [task IAM role in ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)). Please correct me if I am mistaken. I would greatly appreciate any help you can give me. Thank you for your time!

We have a component that starts multiple container on our greengrass device using docker-compose. According to [the documentation](https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html), the correct way to provide the private docker images from ECR is to supply them as artefact: "URI": "docker:public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest" One important aspect of our project is that we can reuse layers of our images. We have limited bandwidth on our devices and images can be big (i.e. PyTorch). If we have modification to do to our images, usually only the last small layer will be updated over the limited bandwidth network. I am worried when we are using images as artifacts, the whole image is going to be re-downloaded. Is that the case ? I currently have a workaround of using the "install" lifecycle step to execute "docker-compose pull" and removing the artifacts, but that has the issue of being run on reboot, while I would rather only have it update the images when there is a new deployment.

Hi I'd like to ask if anyone was able to successfully migrate their existing Multi-container Docker running on 64bit Amazon Linux which is about to be deprecated to the newer ECS running on 64bit Amazon Linux 2? I followed the steps outlined in this document: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/migrate-to-ec2-AL2-platform.html using the AWS CLI to upgrade our environment. It did upgrade it to the new platform however I'm getting a 502 Bad Gateway in my webpage. The document says no configuration needs to be changed on the source code as well as in the Dockerrun.aws.json v2 file. Is there any additional configuration I'm missing? When I rolled-back the platform to the previous one (Multi-container Docker running on 64bit Amazon Linux) - it went back to normal.

I would like to create a scale in policy that uses both cpu utilization and RequestCountPerTarget. for example scale in the task when cpu-utilization < 40% AND RequestCountPerTarget < 100.

I am trying out GreenGrass v2 to see if it would fit our need to deploy our stack using docker-compose to our IoT devices. I have followed the steps [here](https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html) and setup a device with a deployment (i.e. 1.0.1). I am not exactly clear how components interact with docker-compose, it seems that even when I stop a container manually, it will reset on reboot. That is not a problem, but it also looks like when I publish a new deployment (i.e. 1.0.2), I still have containers from 1.0.1 still running, which ends up throwing errors because they are trying to access the same ressource. What is the correct approach to stop the docker-compose stack when I deploy a new update ?

Hi, I followed the wizard to create an ECS/fargate cluster and a basic step function state machine. I was able to run the state machine once (after working through a few permissions issues), though the container exited. I updated the task definition (specifically, all I changed was the container's entrypoint and command), and I'm now encountering a new IAM issue despite not (to my knowledge) changing anything related to the state machine or cluster's roles. ``` Error ECS.AccessDeniedException Cause User: arn:aws:sts::****:assumed-role/StepFunctions-hello-role-****/**** is not authorized to perform: ecs:RunTask on resource: arn:aws:ecs:us-west-2:****:task-definition/hello-task:2 because no identity-based policy allows the ecs:RunTask action (Service: AmazonECS; Status Code: 400; Error Code: AccessDeniedException; Proxy: null) ``` Is there a particular resource that needs to have this role/policy assigned that I'm missing? I don't know how to set or access permissions for "assumed roles" before or after the state machine runs. Thanks!

Hello I need to have a mount point from a volume in my task definition container. the mount point is /example/ex/1:/port ; My question is, is posible to just copy the files in ~/1 to /port instead of overwriting everything in /port ? The other solution will be to make a "sh, -c, ln -s path:path" but i dont know how correct would that be. Thanks

We want to run containers on an ec2 fargate cluster that will run jobs of an arbitrary duration (30 minutes on average). We want to scale out based on active TCP connection metrics via the AWS loadbalancer (1 tcp connection contains stream that needs to be handled by 1 container). This seems possible with cloudwatch metrics. The question is how to **scale in** once the job is done. We do not want to stop a container when it's still processing an active connection. It could take hours for the job to finish (and that is OK). Is there a way we can remove a container once it is done processing (aka no active tcp connection anymore?) If the process inside the container stops, is it removed from the desired active container total?

I am creating this new question, because I could not post an answer on the already existing thread here https://repost.aws/questions/QUiWCpad5jReKxxTiWjKfeyA/how-to-solve-the-ecs-error-youve-reached-the-limit-on-the-number-of-tasks-you-can-run-concurrently I have the same problem as the user in the mentioned thread, but starting an EC2 instance (which I already had before) did not resolve the issue. I can only run 2 tasks concurrently in my ECS Fargate Cluster. The problem is, I don't really know which quota increase request I should choose. I tried `Tasks per service` but so far, I haven't got an answer. I am not sure, but I think there is a bug in the quota view because in the column `Applied quota value` every entry says `Not available` Also, I had to select 5001 as quota (1 above default) which does not make sense to me because I cannot even launch 2 tasks. Maybe someone can help me which this problem.

I have created an ECS Fargate Task, which I can manually run. It updates a Dynomodb and I get logs. Now I want this to run on a schedule. I have setup a scheduled ECS task through EventBridge and through the UI in the ECS cluster. However, this does not run. My looking at the EventBridge logs I can see that the container has been stopped for the following stopped reason: ``` ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.... ``` I thought this might be a problem with permissions. However, I tested giving the Task Execution Role full power user permissions and I still get the same error. Could the problem be something else?

I need to run a container/code that will zip files from S3. I am zipping a lot of files so it could take up to 1 hour to zip all the files which means Lambda is not an option. This leaves me to use Fargate. However, based on my understanding, Fargate will have a pod running at all times. I want to use API Gateway that will "invoke" a Fargate pod, run the code, then terminate the pod when the files are done being zipped. That way each zip function will have its own isolated environment and I will only be charged for the time that I need to zip files and not all the time the pod is running. Is this possible and if so how would one go about it? Or is there another AWS service that would be better suited? Thank you

Hi all, A newbie here. I want to run a small personal Wordpress blog (more like a CV/portfolio page with little content) on AWS. I am interested in learning how Fargate/containers work, that's why I set myself a challenge to run this blog as a container rather than on EC2. I have configured Docker desktop and successfully run the WP container (based on WP and mySQL images) locally. Now my idea is that I need to deploy this container to AWS (Next steps to register a domain via Route 53, etc, but I am not concerned about it now). Questions: 1. I googled first, but most of the proposed solutions for running a Wordpress site in a container on AWS mention also using EC2/RDS/S3. Why? My understanding is that if I run both WP + MySQL in a container, I shouldn't need any other AWS services. --- 2. What would be the costs for running a simple container like that? In the Pricing Calculator one of the parameters is the length of the task, and I'm somewhat confused here. My idea is that I just start the container once and it keeps running till it fails for some reason, and then it just gets restarted. Thank you! Regards E

A customer wants to get various flavors of logs from ECS on EC2, ECS on Fargate, API GW logs, load balancer logs, (and potentially RDS Aurora) to Splunk endpoint Following have already been referenced to the customer Splunk white paper: https://www.splunk.com/pdfs/white-papers/getting-data-into-gdi-splunk-from-aws.pdf  And a couple of these blog posts https://www.splunk.com/en_us/blog/it/splunking-aws-ecs-part-2-sending-ecs-logs-to-splunk.html https://www.splunk.com/en_us/blog/it/splunking-aws-ecs-and-fargate-part-3-sending-fargate-logs-to-splunk.html 
 The challenge is which approach to use as we they ECS Fargate and ECS on EC2 along with other AWS services for which they want to centralize their logs. Currently they are considering separate lambda functions for ECS, lambda for LBs etc. to pull logs from cloudwatch and push them to Splunk endpoint. Trying to seek suggestions on what could be the best practices.

Running a step function state machine accessing an AWS HttpApi to run a lambda using a docker container of ~3 GB size. Fails 20% of the time with #503 error, the rest of the time I get the exact results I expect. It's not a container warm up I believe. Logs for API and lambda (non-container and container) don't show any problems. I have maximized the step functions time out, the retries for the specific lambda, the memory for the lambda (10GB). I have also tried to ping the container and wait 30 seconds in the step functions to no avail. *** Edit: Should have mentioned concurrency for the API is at max (believe it is automatic for HttpApis) Any advice would be appreciated. I don't want to use ECS fargate due to cost and relative complexity. Thanks!

Hi Experts, I need to to change an EFS filesystem from General Performance to Max I/O. The only way to do this is to create a new FS and use DataSync to copy the data. EFS is the underlying storage for my EKS cluster. My approach is to: 1. create the new FS. 2. copy the data with DataSync. 3. Stop services on the EKS cluster. 4. Change the mount points on the CSI Driver. 5. Restart application services. Is the best approach? Or are there other documented steps to change EFS filesystems with an EKS cluster. Thanks in advance.

I'm wanting to send UDP traffic to an EC2 host in the same VPC as an ECS instance and I can only get it to work by testing with allowing all source IPs in the security group for the EC2 instance. With RDS it's always worked fine to use the RDS instance's security group ID as source in the ECS container's security group, but that isn't working here, where I'd use the ECS container's SG as source for the UDP port in the EC2 instance's SG. I tried using something like 172.30.0.0/16 to allow all VPC traffic, but that doesn't work either. Are ECS instances in my VPC? Thanks for any help.

Afternoon all. We have a project that as the title said, the developers will do their magic, then push a docker image up to ECR. The initial test was all manual, ran a few containers on different ports, manually setup the listeners and all good. So the next step it to automate. I want to let them update the ECR/docker image and once done have that update the servers one at a time the way my current code-deploy works. I am trying to read and understand the difference or benefits to using a ECS cluster vs an Elastic Beanstalk as well as understand a bit more from the high level as I am going to need redundancy, autoscaling, etc. So, a brief answer on where to start with what would be great, but basically I need to create a service that starts X amount of docker containers per instance, then they get added to the target group / ELB. Thanks for the advice!

I have created an ECS Fargate Task, which I can manually run. It updates a Dynomodb and I get logs. Now I want this to run on a schedule. I have setup a scheduled ECS task through EventBridge and through the UI in the ECS cluster. However, this does not run. I have used the cron scheduler and I have also tried the 'Run at fixed interval' option, which also doesn't work. As far as possible I am using the same settings when I run the task manually and in EventBridge. I've tried the default capacity provider (and made sure that's set to Fargate in the cluster). I've tried using manually defined permissions and I've tried letting EventBridge make it's own permissions. I see a 'Failed Invocation' in CloudWatch metrics, for every invocation of the task. But I am unable to find any other logs to debug. The task seems to not start: no logs are generated and the failed invocation happens at the same time as the invocation. Are there any logs I could share that would help diagnose? Does anyone have any suggestions for what might be causing this failure?

We are running an S3 Bucket with a Cloudfront In front of it. Recently we've turned enabled block public access (All four options enabled under `Individual Block Public Access settings for this bucket`, but now our Fargate ECS instance is getting `Access Denied` doing a `s3client.putobject`. Images are able to be pulled through the Cloudfront URLs, s3 object are blocked, but we are unable to upload any new images from our ECS instances. How do we allow the ECS task instances to `putObject` A (wide) bucket policy was added: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "1", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::<aws-account>:role/<ecs-task-role>", "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <OAI>" ] }, "Action": "s3:*", "Resource": [ "arn:aws:s3:::<bucket-name>/*", "arn:aws:s3:::<bucket-name>" ] } ] } ``` The ecs-task-role has `AmazonS3FullAccess`. We can manually upload to console. Removing the following two options from block public access does again allow us to put, but then the s3 object urls are public which we want to be non-accessible. - Block public access to buckets and objects granted through new access control lists (ACLs) - Block public access to buckets and objects granted through any access control lists (ACLs)

There is recent linux vulnerability called dirty pipe linux kernel vulnerability i.e CVE-2022-0847. So ECS-optimized Amazon Linux 2 latest AMI does have fix for the latest vulnerable CVE-2022-0847? Please advise. Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html https://www.datadoghq.com/blog/dirty-pipe-vulnerability-overview-and-remediation/

Hi, I have deployed 2 applications in AWS ECS by using Grpc to communicate with each other. However, everytimes the client tried to connect to the server, it complains that the connection is refused. I am using DNS(Route 53) and NettyChannelBuilder to build the connection. I have exposed the client and server port when creating the task. Another thing that I have tried is to create ALB in the server, I also got the same error (Connection refused. No further information). Tried deploying appMesh, it stucks at PRE_INITIALIZING state. May I know what should I do to fix the issue? Note; There is no problem when I deployed the 2 applications in company VM.

We've started experimenting with ECS Anywhere and just in the past few days, the register external instances link is dead/inaccessible: `curl --proto "https" -o "/tmp/ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" ` returns: ``` This XML file does not appear to have any style information associated with it. The document tree is shown below. <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>Q8QPSN1GP90X3ZX2</RequestId> <HostId>DREyWsYwxRMDC1It6G/ThV7gYa6AAzN5U4lO3kfVAzrf2T5A6XPq1ALRVeE6bDSGhmLk/gI/Jug=</HostId> </Error> ```

I'm trying to add awslogs-multiline-pattern to my ECS task definition. For matching timestamp, I'd like to use ``` "awslogs-multiline-pattern": "^\[([0-9]{2}:[0-9]{2}:[0-9]{2} [A-Z]{3})\]" ``` but getting " Unexpected token [ in JSON ..." error during deploy. I couldn't find a direction for a fix/workaround. TIA, Vitaly

I have a build project in AWS CodeBuild. This project uses a docker image stored in AWS ECR. I have to modify the options which are used to run the container - specifically, I want to add `--init`. I see there is a `initProcessEnabled` option which can be used for ECS, but I don't understand how to combine this with CodeBuild.

I'm trying to run a task with an image from an ECR container in the same account. When I try to run the task, it fails, with the following message: "ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::XXXXXXXXX:assumed-rol..." I've tried following this help page: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-unable-to-pull-secrets/ But most of the situations don't apply to me. Also, you will see that the error message is truncated. Do you know how to get the full error message?

by looking at this AWS document: https://docs.aws.amazon.com/step-functions/latest/dg/sample-project-container-task-notification.html step functions can totally instantiate/manage fargate or ECS Task. Speaking about fargate * the example put fargate as first step. suppose fargate is the 2nd step, can we get the output message from first step (larger than 8K) into fargate task through either .sync or .waitForTaskToken model? Instead of passing input parameters through fargate containerOverride's command or commands? * If above is possible, does the code inside fargate suppose to do something like : GetActivityTaskResult getActivityTaskResult = client.getActivityTask(new GetActivityTaskRequest().withActivityArn(stringActivityArn)); String taskToken = getActivityTaskResult.getTaskToken(); String taskInput = getActivityTaskResult.getInput(); * In this step function instantiate fargate model (.sync or .waitForTaskToken), does fargate instances gets created each time when step functions "calls" it? That is, we actually would like to set desire_count and min count both to zero and forget about scaling alarms or metrics. If above is true, does it respect the max_count? I guess cost-wise, it is fine with us because how many concurrent fargate running does not matter anymore. Cost is based on usage. * is ECS or fargate able to return output values to the calling step function? that is, using sendSucess or some other api to send output back to step function for next step to use (become the input of next step)?

My team uses AWS ECS Fargate. We would like to perform a routine recycle for our ECS tasks. I found the MaxInstanceLifetime value parameter but it seems mostly for EC2 instances. Is it possible to set this parameter for ECS Fargate? If so, where in the console can I find this? If not, are there any suggestions on how we can go about recycling our ECS Fargate tasks? Docs: * https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-max-instance-lifetime.html * https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-ec2-auto-scaling-supports-max-instance-lifetime/

Hi, we are experiencing a strange behavior in our pipeline flow. Step from source pulling e image building are correctly marked as succeeded in pipeline but the procedure fall in a "in progress" state for the deploy step. Anyway, when I check ECR panel, I can see the image correctly deployed to cluster. Nothing was changed in the last days about the pipeline setting up, just as usual new commits. Logs are clean and nothing can point me to resolve this issue. Is there something that I can check further to fix this situation? Thanks!

For processing genomics datasets in containers launched from AWS Batch, is it possible to stream the data directly from S3, using an approach similar to SageMaker [pipe mode](https://aws.amazon.com/blogs/machine-learning/using-pipe-input-mode-for-amazon-sagemaker-algorithms/) (or [fast file mode](https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-sagemaker-fast-file-mode/))? Also, are there any hardware limitations that I should be aware of? For example, in [this article](https://aws.amazon.com/premiumsupport/knowledge-center/s3-maximum-transfer-speed-ec2/), the maximum transfer speed between EC2 and S3 is 100 Gbps (VPC endpoints and public IPs in the same region). In a 2018 article, the bandwidth between EC2 and S3 was [stated as 25 Gbps](https://aws.amazon.com/blogs/aws/the-floodgates-are-open-increased-network-bandwidth-for-ec2-instances/)

I have created a brand new AWS account (just to troubleshoot this issue) and the default VPC and subnets of every region in this account are left pristine and unmodified. Here's the default VPC in `us-east-1`: $ aws ec2 describe-vpcs { "Vpcs": [ { "CidrBlock": "172.31.0.0/16", "DhcpOptionsId": "dopt-095a7873b289557a1", "State": "available", "VpcId": "vpc-08ba51697a37c5ad9", "OwnerId": "...", "InstanceTenancy": "default", "CidrBlockAssociationSet": [ { "AssociationId": "vpc-cidr-assoc-0dba5df7b176877b7", "CidrBlock": "172.31.0.0/16", "CidrBlockState": { "State": "associated" } } ], "IsDefault": true } ] } Here's the route table for this VPC: $ aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-08ba51697a37c5ad9 { "RouteTables": [ { "Associations": [ { "Main": true, "RouteTableAssociationId": "rtbassoc-08e6f9833f341f6c4", "RouteTableId": "rtb-000d61d5d0236d276", "AssociationState": { "State": "associated" } } ], "PropagatingVgws": [], "RouteTableId": "rtb-000d61d5d0236d276", "Routes": [ { "DestinationCidrBlock": "172.31.0.0/16", "GatewayId": "local", "Origin": "CreateRouteTable", "State": "active" }, { "DestinationCidrBlock": "0.0.0.0/0", "GatewayId": "igw-0b7ed209f5cd38fa6", "Origin": "CreateRoute", "State": "active" } ], "Tags": [], "VpcId": "vpc-08ba51697a37c5ad9", "OwnerId": "..." } ] } As you can see, the second route permits egress to the internet: { "DestinationCidrBlock": "0.0.0.0/0", "GatewayId": "igw-0b7ed209f5cd38fa6", "Origin": "CreateRoute", "State": "active" } So I assume if I deploy an ECS Fargate task in this VPC, it should be able to pull `amazoncorretto:17-alpine3.15` from `docker.io`. Despite that, whenever I deploy my CloudFormation stack, ECS fails to run the scheduled task as it cannot fetch the images from DockerHub and prints this error: > CannotPullContainerError: > inspect image has been retried 5 time(s): > failed to resolve ref "docker.io/library/amazoncorretto:17-alpine3.15": > failed to do request: Head https://registry-1.docker.io/v2/library/amazoncorretto/manifests/17-alpine3.15: dial ... Here's my CloudFormation template (I have intentionally given wide open permissions to all the roles involved to make sure this issue is not due to insufficient IAM permissions): AWSTemplateFormatVersion: "2010-09-09" Description: ECS Cron Task Parameters: AppName: Type: String Default: CronTask AppImage: Type: String Default: amazoncorretto:17-alpine3.15 AppLogGroup: Type: String Default: ECS AppLogPrefix: Type: String Default: CronTask ScheduledTaskSubnets: Type: List<AWS::EC2::Subnet::Id> Default: "subnet-0031a6eaf7e52173c, subnet-01950a0d2d1e04dc1, subnet-0a1aa70f0421e2025, subnet-036abb95995a86c73, subnet-0f8b5043babfb9a7e, subnet-07cb2210ce2d5bb8f" Resources: Cluster: Type: AWS::ECS::Cluster TaskRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Action: sts:AssumeRole Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Policies: - PolicyName: AdminAccess PolicyDocument: Version: "2012-10-17" Statement: - Action: "*" Effect: Allow Resource: "*" TaskExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Action: sts:AssumeRole Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy Policies: - PolicyName: AdminAccess PolicyDocument: Version: "2012-10-17" Statement: - Action: "*" Effect: Allow Resource: "*" TaskScheduleRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Action: sts:AssumeRole Effect: Allow Principal: Service: events.amazonaws.com Path: / Policies: - PolicyName: AdminAccess PolicyDocument: Statement: - Action: "*" Effect: Allow Resource: "*" TaskDefinition: Type: AWS::ECS::TaskDefinition Properties: Cpu: 256 Memory: 512 NetworkMode: awsvpc TaskRoleArn: !Ref TaskRole ExecutionRoleArn: !Ref TaskExecutionRole Family: !Ref AppName RequiresCompatibilities: - FARGATE ContainerDefinitions: - Name: !Ref AppName Image: !Ref AppImage Command: ["java", "--version"] Essential: true LogConfiguration: LogDriver: awslogs Options: awslogs-create-group: true awslogs-group: !Ref AppLogGroup awslogs-region: !Ref "AWS::Region" awslogs-stream-prefix: !Ref AppLogPrefix TaskSchedule: Type: AWS::Events::Rule DependsOn: - TaskScheduleRole - DeadLetterQueue Properties: Description: Trigger the task once every minute ScheduleExpression: cron(0/1 * * * ? *) State: ENABLED Targets: - Arn: !GetAtt Cluster.Arn Id: ClusterTarget RoleArn: !GetAtt TaskScheduleRole.Arn DeadLetterConfig: Arn: !GetAtt DeadLetterQueue.Arn EcsParameters: LaunchType: FARGATE TaskCount: 1 TaskDefinitionArn: !Ref TaskDefinition NetworkConfiguration: AwsVpcConfiguration: Subnets: !Ref ScheduledTaskSubnets DeadLetterQueue: Type: AWS::SQS::Queue Properties: QueueName: "CronTaskDeadLetterQueue" DeadLetterQueuePolicy: Type: AWS::SQS::QueuePolicy Properties: Queues: - !Ref DeadLetterQueue PolicyDocument: Statement: - Action: "*" Effect: Allow Resource: "*" What am I missing here? Why despite running the task in a public subnet/VPC (below), AWS cannot pull the image from `docker.io`? Is something missing in my `TaskSchedule` resource? TaskSchedule: Type: AWS::Events::Rule ... Properties: ... Targets: - ... EcsParameters: LaunchType: FARGATE TaskCount: 1 TaskDefinitionArn: !Ref TaskDefinition NetworkConfiguration: AwsVpcConfiguration: Subnets: !Ref ScheduledTaskSubnets Thanks in advance.

Hi, I am currently trialing Enhanced Scanning with our ECR repos. What I have identified is, it seems Inspector2 is inferring some information from tags, and using that to detect the OS. For example, see this screenshot -> https://imgur.com/a/VBDNSqL. I have uploaded a container with base image of Oracle Linux 8, with the tag "latest5". ECR is taking the "5" as the OS version, and treating this container as Oracle Linux 5. You can see in the screenshot also, the CVE's report, and the OS they affect are RHEL 5/6. I am trying to understand if this is expected behavior or a bug in ECR? Cheers!

Getting Connection refused by public ip address, although image pulled from ECR running appropriately in pod container. Security Group >>Inbound Group >>Type: HTTP, SSH, TCP, Port: 80, 22, 0-65535 all allowed for IPv4, even assigned Elastic IP to NAT Gateway which is allowing Public Ip through Internet Gateway

Hi, I am currently trying to create a custom metric based on CPUUtilization for faster autoscaling. The one provided by AWS with 1-min resolution is not good enough. While finding a solution on how to create one, I came across this blog post which shows using System Manager and unified Cloud Agent to create a scaling policy using Memory Utilization - [BLOG](https://aws.amazon.com/blogs/mt/create-amazon-ec2-auto-scaling-policy-memory-utilization-metric-linux/). I checked and found that there is no metric collected by the CloudWatch Agent under the name *CPUUtilization* but there is one under the name *cpu_usage_active*. So, I am confused as to whether both the metrics are same or different. If different, then how so? Thanks

Our application is deployed on fargate and it uses EFS to store files. We are experiencing an issue where [the written files don't show up because of NFS negative caching](https://repost.aws/questions/QU0j0fRjfWTGG5BSHuGVfxCg/aws-efs-file-delete-and-recreate-not-detected-programmatically-for-25-to). The [solution](https://repost.aws/questions/QU0j0fRjfWTGG5BSHuGVfxCg/aws-efs-file-delete-and-recreate-not-detected-programmatically-for-25-to) seems to be using `noac` or `lookupcache=pos` mount options for EFS. But I don't see a way to specify mount options when configuring EFS as a volume on Fargate. Is there a solution for Fargate or a workaround?

Hello, I am setting up a Yellowfin [deployment](https://wiki.yellowfinbi.com/display/yfcurrent/Install+in+a+Container) using their stock app-only [image](https://hub.docker.com/r/yellowfinbi/yellowfin-app-only) on ECS Fargate. I was able to set up a test cluster for my team to experiment with. Yellowfin requires a license to use their software. To issue a license, Yellowfin needs to know the hostname of the underlying platform it runs on. Yellowfin can provide wildcard licenses that match on a standard prefix or suffix. Currently, we are using a development license that matches on the default hostname that our test environment's Fargate task is assigned. The default hostname seems to be of the form <32-character alphanumeric string>-<10-digit number> where the former is the running task's ID and the latter is an ID of some other related AWS resource (the task definition? the cluster?) that I could not identify. Although this 10-digit number stays constant if new tasks are run, it does not seem like a good strategy to base the real Yellowfin license off of it. I would like to override the hostname of a Fargate task when launching the container to include a common prefix (e.g., "myorg-yfbi-") to make it simple to request a wildcard license for actual use. If possible, I would like to avoid building my own image or migrating to use another AWS service. Is there a standard way to override the hostname for a Fargate service by solely updating the task definition? Would overriding the entrypoint be a viable option? Is there another way to set hostname in Fargate that I am not aware of? Thank you for any guidance you can provide. Happy to provide more information if it helps.

If I push a new tag for an existing image in ECR, it is shown in the UI as a separate image. Likewise I may have many different images, where only the top layer changes, but it will be shown as a lot of (large) images. Is there a way to view the actual storage utilization of the account on ECR, taking into account the layers that are shared between images? Thank you for your time

The "new" Amazon ECS Console, which is an opt-in experience, is missing important functionality. One such missing function is the ability to specify a custom entrypoint or command override for each container defined in an Amazon ECS Task Definition. This makes the usage of the "new" console almost impossible, and I have to fallback to using the old console. When is the "new" ECS console going to be usable?

I have a customer moving AWS Workloads to AWS - AMS (Amazon Managed Services) most of these services are hosted on ECS and Lambda, is there an assessment questionnaire to help us assess the effort/cost and complexity of these applications?

Hello. I need to calculate the PDF task on my Fargate cluster and after finishing calculating the task should be stopped or terminated to decrease payments. The consequence of my actions: 1. Little task always running on EC2 Cluster and check DB to the new data. 2. If new data appears, using Boto3, run the Fargate container (big task to calculate PDF). 3. After the job is done, the task should be stopped. Also if in the DB appears the second row of data during proceeding the first task, Fargate should create a second task for the second job. and then stop tasks... So, I have: * PDF task written on Python and deployed on ECR * Fargate cluster * Task definition (describe Memory, CPU's and container) What is my next step? How task know, that it should be closed? Any idea or solution?

I'm using the AWS console to create an ECS service (using fargate) in an existing cluster. In the second step of the wizard (configure network) I choose an existing application load balancer. The "container to load-balance" section shows my container to add. When I click "add to load balancer" it initially shows the "production listener port" and "target group name" dropdowns showing "create new". When I select an existing target group in the dropdown this grays out (disables) the "production listener port" dropdown. When clicking the "next step" button validation complains the "production listener port" is not filled in (validation message: "please select a listener"). Which isn't possible because the control is disabled. First selecting a listener port in the wizard and switching to an existing target group after that doesn't remedy the situation as choosing an existing target group blanks out and disables the "production listener port" dropdown causing the same problem when clicking the "next step" button. How to get the container to register in an existing target group? **Update** The target group is an empty IP address group. Interestingly it does work if the target group is not empty (the "production listener port" is then filled with 443:HTTPS) but an empty target group (even of the correct type) clears the "production listener port". **Reproduction** 1. Create a new target group of type "IP address" leaving other default settings. Do not register any targets in this group. 2. Next add this target group to a (new or existing) load balancer (for testing I added the group to an existing load balancer with a single source IP address filter e.g. 192.168.1.1/32 so it doesn't disrupt normal operations). 3. When creating a new ECS service select a task definition that has a container with an exposed port 80. 4. On the second screen "Configure Network" choose the VPC and subnets and under "Load balancing" pick "Application load balancer". 5. Select the load balancer to which you added the empty target group. 6. Now click "Add to load balancer" next to the container. This shows the "Production listener port" and "Target group name" dropdowns both initially set to "Create new". 7. Choosing the empty target group disables the "Production listener port" dropdown and clears it. 8. If the target group is not empty the "Production listener port" is automatically filled with the correct port.

Hi All, We are trying to migrate from EC2 to AWS ECS Fargate and this is a PCI complaint environment. We would like to get inputs on how we could be complaint for PCI with Fargate implementation. Particularly it would be helpful if I get some inputs on how everyone is dealing with the run-time security of Fargate containers. It would be nice to know as to how requirements like anti-virus, FIM which were in EC2 are covered with AWS Fargate. Regards, Karthik

I have a distributed application set up in a Docker environment. Currently, I am able to run the application in my local machine, using docker-compose to start both the manager and the workers. The **docker-compose.yaml** file looks similar to this: ``` version: "3" services: manager: image: ${IMAGE} command: binary start-manager --port 1234 worker: image: ${IMAGE} command: binary start-worker --address manager --port 1234 depends_on: - manager deploy: replicas: 2 ``` This application is meant to run once, and when the manager finishes, it stops the workers and then stops itself. Currently, I am able to run the application in AWS using a docker context (I followed [this](https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/)). The main problem I have is that once it is run using `docker --context myContext compose up`, it creates ECS services that, once the specific task finishes, it creates a new task; so I have to monitor the containers and run `docker --context myContext compose down` when it finishes. I found that *service.deploy.restart_policy* is not implemented, so my question is: can I run this application once, and ensure that no tasks are spawned again once the manager service stops? Thanks, Josep

Hi All, I am currently understanding the basics of ECS Auto Scaling and am confused how many load balancers of what schemes ("internal" or "internet-facing") are required to create a cluster which is ASG aware using capacity provider. If I have configured a load balancer in service (scheme-'internet facing' with target group 'IP address'), Do I need to configure a load balancer in Auto Scaling Group? If Yes, then of what schemeand target group? My understanding though not tested is when both the load balancers are configured, the LB of ASG, let's say ASG-LB, will be 'internet-facing' with target group of 'instances'. The LB of Service, let's say S-LB, will be 'internal' with target group of 'IP Address'. When a request is send to the cluster, ASG-LB will direct the request of a particular instance depending on the traffic and S-LB will direct the request to a particular task inside of that instance. I would like to know whether I am correct? If not, then please help me understand how the traffic is routed in the cluster with this two load balancers? Thanks, Priyam Mehta

I am trying to query the AMI by lookup up the AMI Alias in SM Parameter Store for a specific version of Amazon ECS-Optimized Amazon Linux 2 AMI. The AMI Alias listed on Marketplace (https://aws.amazon.com/marketplace/server/configuration?productId=9c5eb799-bb6a-4a52-8fda-8e90b3f751f1) is `Ami Alias: /aws/service/marketplace/prod-phaazl5qm6pka/2.0.20220121` When I query `Name: equals: /aws/service/marketplace/prod-phaazl5qm6pka/2.0.20220121` I get "You do not have any parameters in this region." I am able to query the "latest" version of the AMI Alias with a query `Name: equals: /aws/service/marketplace/prod-phaazl5qm6pka/latest` and get back exactly on result that has a string value with the expected AMI (currently `ami-0e4efed85dffc2b28` for `us-east-2`). I want to create some automation that uses the AMI Alias for specific version, and not "latest" and be able to lookup the AMI by region. What am I doing wrong in my query? Or am I maybe missing some permissions or something somewhere?

We have an AWS Batch compute environment that is set up to use EC2 On-demand instances. Whenever we run a large number of tasks, Batch scales our group of instances up to the number of CPUs we require, but it never ends up scaling them back down, even when we have 0 Batch tasks running. Looking at the autoscaling group that Batch manages, I see it has scale-in protection enabled. This means it doesn't terminate EC2 instances when the desired capacity is decreased. So - looking at the autoscaling group's logs, it always scales up correctly, but scaling down fails because of the scale-in protection. For example: ``` Status: CANCELLED Error: Could not scale to desired capacity because all remaining instances are protected from scale-in. Cause: At 2022-02-02T13:11:53Z a user request update of AutoScalingGroup constraints to min: 0, max: 18, desired: 18 changing the desired capacity from 96 to provide the desired capacity of 18. At 2022-02-02T13:12:05Z group reached equilibrium. ``` My guess is this is why it isn't scaling down. But since this auto scaling group is managed by Batch and Batch should scale instances down relatively quickly, it doesn't make sense to me. This is costing us a lot in unused EC2 time, so if anyone can offer any insights, that'd be great.

Hi, There is a weird problem statement. So I have a ECS Task attached with RoleName called `MyRole` . This ECS Tasks has a nodejs script which queries a dynamodb Table. Nodejs script: ```js const AWS = require('aws-sdk'); async function GetCredentials(){ var docClient = new AWS.DynamoDB.DocumentClient({region: 'us-east-2'}); var identifier = "test-asassa"; var params3 = { TableName: 'MyTable', ExpressionAttributeNames: { '#var1': 'var1' }, KeyConditionExpression: '#var1 = :var1', ExpressionAttributeValues: { ':var1': 'ValueToBeQueried' } } await docClient.query(params3,( () => { })).promise().then(async (data1) => { console.log(data1); }); }; GetCredentials().then(data => { console.log(data); }); ``` This code when ran simply in docker container doesnt fetch the AWS credentials from Role and errors out saying `ValidationException`. At same time, if I export the credentials as `AWS_ACCESS_KEY_ID` , `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN`, the script works. Can anyone help me understand wth is going on here? How do i make sure code takes the role attached to the ECS Task? Trust relationship and all other misc are correct. The query is also correct. To reproduce this, you can create a table called "MyTable" with primary key as "var1" . Now use the same code as shared above and in the same terminal, export the AWS access keys. The code will work. Now, in any ecs task, attach iam role to the task with sufficient permissions and it will not work and error out saying `ValidationException: One or more parameter values were invalid: Condition parameter type does not match schema type` which is completely mis-leading error.

I need to deploy an ML app that needs GPU access for its response times to be acceptable (since it uses some heavy networks that run too slowly on CPU). The app is containerized and uses an nvidia/cuda base image, so that it can make use of its host machine's GPU. The image alone weighs ~10GB, and during startup it pulls several ML models and data which takes up about another ~10GB of disk. We were previously running this app on Elastic Beanstalk, but we realized it doesn't support GPU usage, even if specifying a Deep Learning AMI, so we migrated to ECS, which provides more configurability that the former. However, we soon ran into a new problem: **selecting a g4dn instance type when creating a cluster, which defaults the AMI to an ECS GPU one, turns the Root EBS Volume Size field into a Data EBS Volume Size field.** This causes the instance's 22GB root volume (which is the only one that comes formatted and mounted) to be too small for pulling our image and downloading the data it needs during startup. The other volume (of whatever size I specify during creation in the new Data EBS Volume Size field) is not mounted and therefore not accessible by the container. Additionally, the g4dn instances come with a 125GB SSD, that is not mounted either. If either of these were usable or it was possible to enlarge the root volume (which it is if using the default non-GPU AMI) ECS would be the perfect solution for us at this time. At the moment, we worked around this issue by creating an *empty* cluster in ECS, and the manually creating and attaching an Auto Scaling group to it, since when using a Launch configuration or template the root volume's size can be correctly specified, even if using the same exact ECS GPU AMI as ECS does. However, this is a tiresome process, and makes us lose valuable ECS functionality such as automatically spawning a new instance during a rolling update to maintain capacity. Am I missing something here? Is this a bug that will be fixed at some point? If its not, is there a simpler way to achieve what I need? Maybe by specifying a custom launch configuration to the ECS cluster or by automatically mounting the SSD on instance launch? Any help is more than appreciated. Thanks in advance!

Using AWS CodePipeline and setting a Source, Build and passing `taskdef.json` and `appspec.yaml` as artifacts, the deployment action `Amazon ECS (Blue/Green)` will fail with the error: STRING_VALUE can not be converted to an Integer This error does not specify where this error happens and therefore it is not possible to fix. For reference, the files look like this: ```yaml # appspec.yaml version: 0.0 Resources: - TargetService: Type: AWS::ECS::Service Properties: TaskDefinition: <TASK_DEFINITION> LoadBalancerInfo: ContainerName: "my-project" ContainerPort: 3000 ``` ```json // taskdef.json { "family": "my-project-web", "taskRoleArn": "arn:aws:iam::1234567890:role/ecsTaskRole-role", "executionRoleArn": "arn:aws:iam::1234567890:role/ecsTaskExecutionRole-web", "networkMode": "awsvpc", "cpu": "256", "memory": "512", "containerDefinitions": [ { "name": "my-project", "memory": "512", "image": "01234567890.dkr.ecr.us-east-1.amazonaws.com/my-project:a09b7d81", "environment": [], "secrets": [ { "name": "APP_ENV", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:APP_ENV::" }, { "name": "PORT", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:PORT::" }, { "name": "APP_NAME", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:APP_NAME::" }, { "name": "LOG_CHANNEL", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:LOG_CHANNEL::" }, { "name": "APP_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:APP_KEY::" }, { "name": "APP_DEBUG", "valueFrom": "arn:aws:secretsmanager:us-east-1:1234567890:secret:web/my-project-NBcsLj:APP_DEBUG::" } ], "essential": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "", "awslogs-region": "", "awslogs-stream-prefix": "" } }, "portMappings": [ { "hostPort": 3000, "protocol": "tcp", "containerPort": 3000 } ], "entryPoint": [ "web" ], "command": [] } ], "requiresCompatibilities": [ "FARGATE", "EC2" ], "tags": [ { "key": "project", "value": "my-project" } ] } ``` Any insights on this issue are highly appreciated!

``` public class Example { public static void main(String[] args) { // print statement at the start of the program System.out.println("Start..."); System.out.print("Number of available processors are: "); System.out.println( Runtime.getRuntime().availableProcessors()); } } ``` Ref: https://www.tutorialspoint.com/get-number-of-available-processors-in-java When I compile and run this code on AWS ECS Fargate getting "1" as the output even though I have specified more than 1 cpu in task def(4098 for 4 vCPU). **Tested Java versions:** - Corretto-17.0.0.35.1 - Corretto-8.275.01.1 But when I do the same on AWS EC2 inside docker by allocating CPU with --cpus for "docker run" I get the exact number what I pass to --cpus. Same results with AWS EC2 with Containerd as well. Also tested this in EKS 1.18 by setting CPU limit. It returns the number of CPUs specified in the CPU limit. So, wondering why "1" only in AWS ECS.

Hi, We are having two microservices running in separate containers in ECS cluster, we are getting "connection reset" error in microservice when it tries to call another microservice over REST. We are using Elastic Load Balancer and we verified that idle timeout on ELB is sufficiently large. Does anyone faced this issue before? Any pointers will be helpful. Thank You in advance. Vishwas

I'm new to Cloud/DevOps stuff. I have dockerized a wordpress website where site runs on a one container and mysql db running on a different container. I have the code hosted on github. This I imagine to be a high volume website and I need to be highly available and horizontally scalling. Also I want the application to be in a pipeline so that when I make changes to the code, the updates get automatically pushed to the running website. I want to host the application on AWS. Being a student, I have little idea in setting this up on AWS. Could anyone help me or direct me to the right direction?

I have an ECS service which is of Launch Type EC2 owned by an AWS account A. Our IT team has created an FSx storage owned by an AWS Account B - [see simple diagram here](https://i.stack.imgur.com/MyU1d.png) When I try to launch tasks I get this error in the Stopped reason section of the task: ``` Stopped reason Fsx describing filesystem(s) from the service for [fs-0c52aba0aac20c744]: FileSystemNotFound: File system 'fs-0c52aba0aac20c744' does not exist. ``` I have attached those 2 policies to the EC2 (container host) instance: - AmazonFSxReadOnlyAccess (AWS Managed) - fsx_mount (Customer Managed) fsx_mount: ``` { "Statement": [ { "Action": [ "secretsmanager:GetSecretValue" ], "Effect": "Allow", "Resource": "arn:aws:secretsmanager:us-west-2:111111111111:secret:dev/rushmore/ad-account-NKOkyh" }, { "Action": [ "fsx:*", "ds:DescribeDirectories" ], "Effect": "Allow", "Resource": "arn:aws:fsx:us-west-2:222222222222:file-system/fs-0c52aba0aac20c744" } ], "Version": "2012-10-17" } ``` **Note** that the account id of 222222222222 represents AWS Account B. Also, **VPC Peering is in place between the EC2 instance VPC and the FileSystem VPC**. Terraform aws_ecs_task_definition: ``` resource "aws_ecs_task_definition" "participants_task" { volume { name = "FSxStorage" fsx_windows_file_server_volume_configuration { file_system_id = "fs-0c52aba0aac20c744" root_directory = "\\data" authorization_config { credentials_parameter = aws_secretsmanager_secret_version.fsx_account_secret.arn domain = var.domain } } } ... } ``` I am not sure why ECS cannot find the FSx file system. Surely it must be because it is in another AWS account but I don't know what changes are required in order to fix this.

After running the command ``` docker push 123456.dkr.ecr.us-west-2.amazonaws.com/myapp ``` I then see ``` 111111 Retrying 222222 Retrying ... 444444 Waiting ... EOF ``` I know that I authenticated to AWS through the AWS CLI. What could be blocking my attempts to push my Docker image to ECR?

It is annoying to see #Fargate tasks rebooting every six hours and there is no documentation. (I have seen this in the past). Web search finds one relevant personal blog post: https://stephengream.com/fargate-task-recycling and the reference to the AWS article (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-recycle.html) is broken.  \[See the screenshot here: https://i.imgur.com/tfGZ7IK.png because AWS doesn't render it for god knows why!] Neither Google Cache nor http://Archive.org has it! Any idea why Fargate tasks reboots every 6 hours? What is Task Recycle?

As part of a conversation about whether or not we need end to end TLS, I am trying to understand the **communication path between an ALB and the ECS Fargate tasks it targets**. I'm assuming the ALB and service tasks are hosted on completely distinct instances? Are these instances logically part of my VPC or not? Is the traffic confined to my VPC or does it go via AWS network's? I am arguing there is nothing to gain with TLS between ALB and services, as long as we trust AWS network, but I cannot find official document to support my argument, other than a SO thread from a AWS employee. Can anyone shed some light? Thanks