Questions tagged with Parameter Store

We have an ECS cluster in us-west-2 that runs a few ECS services. We run some ECS tasks that are invoked periodically via EventBridge. All tasks use the EC2 launch type and run on container instances that we manage with an Auto Scaling Group. AMI used currently is amzn2-ami-ecs-hvm-2.0.20220630-x86_64-ebs. Container instances are launched in private subnets and VPC endpoints are set up for a few AWS services, including SSM. A few months ago we started seeing missed checkins from the periodically launched tasks and saw that at least some of them failed to launch due to a timeout from the SSM API endpoint. In ecs-agent's log, it shows up like: > level=error time=2022-09-19T22:30:56Z msg="Failed to create task resource" error="fetching secret data from SSM Parameter Store in us-west-2: RequestError: send request failed\ncaused by: Post \"https://ssm.us-west-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" task="..." resource="ssmsecret" > level=info time=2022-09-19T22:30:56Z msg="Setting terminal reason for task" reason="fetching secret data from SSM Parameter Store in us-west-2: Request Error: send request failed\ncaused by: Post \"https://ssm.us-west-2.amazonaws.com/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" task="..." We tried increasing the throughput of SSM Parameter Store through its settings, but it didn't seem to have an effect. https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-throughput.html Other guides and Q&As I could find were about network misconfigurations that would lead to a complete inability to talk to SSM, whereas the symptom I'm seeing is only intermittent; the ECS tasks get launched without an issue most of the time. https://aws.amazon.com/premiumsupport/knowledge-center/ssm-tcp-timeout-error/ What could be the cause? What else can I look into?lg...

I’m currently working on a java project that will be deployed to elastic beanstalk and storing all the sensitive properties to parameter store. My problem is I need to restart the App server on elastic beanstalk after the deployment to get those parameters to work. I need that to be automatically picked up or automatically restart the app server. Example: I need to store db access details in parameter store and make them available in environment properties in elastic beanstalk and I have done that by adding a bash script file in .platform folder to get the parameters and add them to env files /opt/elasticbeanstalk/deployment/env  Referenced link: https://www.fullstackerconsulting.com/2021/09/09/how-can-i-use-the-aws-systems-manager-parameter-store-with-an-aws-elastic-beanstalk-instance-to-manage-environment-variables/ Aws: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html For demo App, I'm using java to get the properties through “System.getenv("dbusername");” The scripts are running and I’m able to access the parameter store properties but the problem is I need to restart the App server on elastic beanstalk to work these properties. Can you please someone suggest to me that I can run the App server automatically once the application is deployment is done? Let me know if you need any further information related to my issue.lg...

Hi, As I updated AMI ID in parameter store to the latest untested version to try to test it on our Integration and UAT environments on some infrastructure codepipeline, then it turned out that we needed to change the AMI back to the previous tested version to do another codepipeline deployment on integration, UAT and production, so the deployment was successful. I then had to change the AMI ID back to the latest version in the parameter store so that we can test another infrastructure codepipeline for Integration and UAT, I then got an error in one of the ECS cloudformation stack during the deployment on integration as "UPDATE_ROLLBACK_COMPLETE" Can anyone help where I should check to find the root cause please? Many thanks in advance Dellg...

My app uses SSM Parameter Store on Fargate instances and locally in a Docker container. We're accessing it with Boto3 from Python. Multiple developers on my team, in different countries, have seen a very intermittent issue, cropping up maybe once every 1–4 weeks, where for 10 minutes or so, calls to SSM will fail with this error: ``` botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://ssm.us-east-2.amazonaws.com/" ``` The ECS instances do not see the issue as far as I'm aware, this is only a problem when we're accessing the endpoint via Boto3 from our home networks. It occurs to me now that I haven't verified whether all users see the problem at the same time, or if it's just one user at a time. I will try to test this the next time I see it. I have tried: 1. Reducing the number of calls we make to SSM. It's now down to about 2/sec per user at the maximum, with effectively no other users cuncurrently hitting the API. So we're never getting anywhere near the [40 requests/second limit](https://docs.aws.amazon.com/general/latest/gr/ssm.html#limits_ssm). In looking at the logs, the most I can see is 12 requests in *one minute.* We're just not using this very agressively, so it doesn't seem possible that the problem is throttling. All of our calls are paginated calls to GetParametersByPath, and we are using `WithDecryption=true`. 2. Changing the Boto3 retry method from Legacy to [Standard](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#standard-retry-mode). This is probably a good thing to do anyway, but doesn't seem to have fixed the problem. The only reliable solution I've come up with is to wait. Eventually, the endpoint comes back and my application begins working again. But this is really an unacceptable level of service interruption, and I feel like I must be doing something wrong. Is there a setting I have overlooked? Does anyone have any troubleshooting suggestions for things to try when I inevitably see the problem again?lg...

I am trying to enable X-Ray on our application. At first we were only getting information for Dynamo, but then I realized I needed to include the instrumentors for both v1 and v2 versions of the dependencies in maven. Now my problem is that I am getting tons of errors because segments aren't created for initialization tasks. ``` 10:46:35.602 [main] ERROR o.s.c.a.p.AwsParamStorePropertySourceLocator - Fail fast is set and there was an error reading configuration from AWS Parameter Store: Failed to begin subsegment named 'AWSSimpleSystemsManagement': segment cannot be found. 10:46:35.645 [main] ERROR o.s.boot.SpringApplication - Application run failed ``` I found an article https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-startup.html that seemed like it would give me what I wanted. The problem is that in that article the person has access to the code that is calling into the AWS SDK. In my case during start up we are calling to get a parameter store. This all happens outside of my direct control so I cannot easily wrap the call in a begin and endSegment call. I am using Java 11, Springboot, and Maven. I could just ignore the errors or change the logging level of that particular class to mask the issue, but that feels hacky. Is there another way to solve this? Any examples anywhere of someone solving it?lg...

The task is simple: whenever an EC2 instance is launched with tag key:value I want it to install a specific software. Whenever an EC2 instance is launched with a different tag key:value I want it to install a different software. I understand that I can create 2 different associations in State Manager that uses runCommand RuneRemoteScript to install software based on the tags, but the goal is to have 1 composite document that can do this. Any help / guidance would be appreciated!lg...

Good evening to all, when I edit an xml configuration file under path **C:\appname** during the session, when I disconnect this change is lost even if I have enabled in the stack the **application persistance setting** and the **home folder S3** in storage. Do you have any advice? Thank you!!lg...

I have replicated my environment using the same information with respect to SSM , S3 , parameter store , secrets manager, I did not change any values, I am getting the below error when I run my workflow ( github actions ) ca any please help what could be the issue. Unhandled exception. Amazon.SimpleSystemsManagement.AmazonSimpleSystemsManagementException: The parameter doesn't meet the parameter name requirements. The parameter name must begin with a forward slash "/". It can't be prefixed with \"aws\" or \"ssm\" (case-insensitive). It must use only letters, numbers, or the following symbols: . (period), - (hyphen), _ (underscore). Special characters are not allowed. All sub-paths, if specified, must use the forward slash symbol "/". Valid example: /get/parameters2-/by1./path0_.lg...

Hi everyone! Is there a way to inject a Parameter Store parameter into a Docker label value? That can be done for the environment and logging variables via the `valueFrom` option but the Docker label values are ordinary strings, and I would like to avoid creating a new task definition just to update the e.g. version. Any thoughts and ideas are appreciated!lg...

Hi, I'm looking to see what options are available and what might be the best practice to accomplish routing HTTP traffic based on a parameter within the parameter store. Currently we have multiple services all sending HTTP traffic to a single URL. This receiving endpoint is about to be duplicated and we will be introducing a setting in the parameter store that will determine whether traffic will be send to endpoint A, or endpoint B. A lambda function is always an option, but I'm looking to see if there's anything already within the managed services side of AWS that can help us. At first ALB seemed a logical choice as that has routing rules, unfortunately it seems the rules can only be determined by attributes of the incoming request, and not a Parameter from the parameter store. So perhaps API gateway could be utilised, but with around a million requests per day, I'm concerned this would be costly and it's also not clear whether I forward traffic based on a setting from the parameter store. Any thoughts appreciated, thanks!lg...

Child1SG stack: AWSTemplateFormatVersion: 2010-09-09 Description: Basic SSH Child Security Group. Resources: \ MySSHSecurityGroup1: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: my new SSH Child SG SecurityGroupIngress: - IpProtocol: tcp - FromPort: '22' - ToPort: '22' - CidrIp: 0.0.0.0/0 Root Stack: AWSTemplateFormatVersion: 2010-09-09 Description: Parameter store of SG1 Resources: myStackWithParams: Type: AWS::CloudFormation::Stack Properties: **TemplateURL: https://cf-templates-o3rmx0wkf9l-us-east-1.s3.amazonaws.com/2022115bE6-Child1SG.yaml #arn** For this template URL how to pass dynamically S3 path of child .lg...

Child1SG stack: AWSTemplateFormatVersion: 2010-09-09 Description: Basic SSH Child Security Group. Resources: MySSHSecurityGroup1: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: my new SSH Child SG SecurityGroupIngress: - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 0.0.0.0/0 Root Stack: AWSTemplateFormatVersion: 2010-09-09 Description: Parameter store of SG1 Resources: myStackWithParams: Type: AWS::CloudFormation::Stack Properties: TemplateURL: https://cf-templates-o3rmx0wkf9l-us-east-1.s3.amazonaws.com/2022115bE6-Child1SG.yaml #arn For this **template URL** how to pass dynamically S3 path of child .lg...