Hi Team,
I'm trying to use AWS Batch service with ECS Fargate. It's basically a python script to fetch the db password stored as secret from AWS SSM Parameter and run an ETL function.
I have ensured networking(internet access with NAT Gateway) and the required iam permission(Full Access) to fetch the secrets or ecr image. It is scheduled to run on an hourly basis. Sometimes, it is working fine but some other time it is failing with the below message.
"Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secrets from ssm: service call has been retried 5 time(s): RequestCanceled: request context canceled caused by: context deadline exceeded"
This seems to be a strange issue. I'm happy to fix if any changes to be done from my side but i'm little worried on why it is unstable. Can some clarify on this issue please?
Since it is an intermittent issue, it seems to be related to the subnet configuration (NACL?). Is it possible to share the networking configuration that you are using? e.g.: subnet conf, route table, SG, NACL, etc?
All this would help to troubleshoot this. "context deadline exceeded" is usually a network issue. Also, if you have a Premium Support plan, I'd encourage you to open a case with them to get more help.
@Henrique Thanks for the response. We are using 2 private subnets for running AWS Batch fargate container. NACL is opened for all traffic ingress and egress. Route table is configured with Nat Gateway for Internet traffic. Egress of SG is opened to all traffic and ingress is empty. We do not have premium support. I have been running it as a cron schedule for every 2 hours. Success rate is around 70% and it is getting failed with the same error other times. Can you please let me know what else the issue maybe? Will this issue be fixed if i use AWS Secret manager instead of SSM Parameter store?
"context deadline exceeded" errors are likely a network issue. We have this knowledge center article that might help you: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-unable-to-pull-secrets/ Let me know if this helps you