By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Parameter Store

Sort by most recent
  • 1
  • 2
  • 12 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

ECS agent sporadically times out while fetching secrets from SSM Parameter Store

We have an ECS cluster in us-west-2 that runs a few ECS services. We run some ECS tasks that are invoked periodically via EventBridge. All tasks use the EC2 launch type and run on container instances that we manage with an Auto Scaling Group. AMI used currently is amzn2-ami-ecs-hvm-2.0.20220630-x86_64-ebs. Container instances are launched in private subnets and VPC endpoints are set up for a few AWS services, including SSM. A few months ago we started seeing missed checkins from the periodically launched tasks and saw that at least some of them failed to launch due to a timeout from the SSM API endpoint. In ecs-agent's log, it shows up like: > level=error time=2022-09-19T22:30:56Z msg="Failed to create task resource" error="fetching secret data from SSM Parameter Store in us-west-2: RequestError: send request failed\ncaused by: Post \"https://ssm.us-west-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" task="..." resource="ssmsecret" > level=info time=2022-09-19T22:30:56Z msg="Setting terminal reason for task" reason="fetching secret data from SSM Parameter Store in us-west-2: Request Error: send request failed\ncaused by: Post \"https://ssm.us-west-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" task="..." We tried increasing the throughput of SSM Parameter Store through its settings, but it didn't seem to have an effect. https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html Other guides and Q&As I could find were about network misconfigurations that would lead to a complete inability to talk to SSM, whereas the symptom I'm seeing is only intermittent; the ECS tasks get launched without an issue most of the time. https://aws.amazon.com/premiumsupport/knowledge-center/ssm-tcp-timeout-error/ What could be the cause? What else can I look into?
0
answers
0
votes
13
views
asked 6 days ago

Need automatic app restarts in Elastic BeanStalk after completing the deplyoment with using .platform

I’m currently working on a java project that will be deployed to elastic beanstalk and storing all the sensitive properties to parameter store. My problem is I need to restart the App server on elastic beanstalk after the deployment to get those parameters to work. I need that to be automatically picked up or automatically restart the app server. Example: I need to store db access details in parameter store and make them available in environment properties in elastic beanstalk and I have done that by adding a bash script file in .platform folder to get the parameters and add them to env files /opt/elasticbeanstalk/deployment/env ![Enter image description here](/media/postImages/original/IM9nY14CVuRE-aFek6wnWL7Q) Referenced link: https://www.fullstackerconsulting.com/2021/09/09/how-can-i-use-the-aws-systems-manager-parameter-store-with-an-aws-elastic-beanstalk-instance-to-manage-environment-variables/ Aws: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html For demo App, I'm using java to get the properties through “System.getenv("dbusername");” The scripts are running and I’m able to access the parameter store properties but the problem is I need to restart the App server on elastic beanstalk to work these properties. Can you please someone suggest to me that I can run the App server automatically once the application is deployment is done? Let me know if you need any further information related to my issue.
0
answers
0
votes
11
views
asked a month ago

Intermittent ConnectTimeoutError accessing SSM

My app uses SSM Parameter Store on Fargate instances and locally in a Docker container. We're accessing it with Boto3 from Python. Multiple developers on my team, in different countries, have seen a very intermittent issue, cropping up maybe once every 1–4 weeks, where for 10 minutes or so, calls to SSM will fail with this error: ``` botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ssm.us-east-2.amazonaws.com/" ``` The ECS instances do not see the issue as far as I'm aware, this is only a problem when we're accessing the endpoint via Boto3 from our home networks. It occurs to me now that I haven't verified whether all users see the problem at the same time, or if it's just one user at a time. I will try to test this the next time I see it. I have tried: 1. Reducing the number of calls we make to SSM. It's now down to about 2/sec per user at the maximum, with effectively no other users cuncurrently hitting the API. So we're never getting anywhere near the [40 requests/second limit](https://docs.aws.amazon.com/general/latest/gr/ssm.html#limits_ssm). In looking at the logs, the most I can see is 12 requests in *one minute.* We're just not using this very agressively, so it doesn't seem possible that the problem is throttling. All of our calls are paginated calls to GetParametersByPath, and we are using `WithDecryption=true`. 2. Changing the Boto3 retry method from Legacy to [Standard](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#standard-retry-mode). This is probably a good thing to do anyway, but doesn't seem to have fixed the problem. The only reliable solution I've come up with is to wait. Eventually, the endpoint comes back and my application begins working again. But this is really an unacceptable level of service interruption, and I feel like I must be doing something wrong. Is there a setting I have overlooked? Does anyone have any troubleshooting suggestions for things to try when I inevitably see the problem again?
1
answers
0
votes
68
views
asked 2 months ago
  • 1
  • 2
  • 12 / page