By using AWS re:Post, you agree to the Terms of Use

Questions tagged with AWS Systems Manager

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

SSM Agent update problems

I'm having issues with SSM for the last 3 days with the RUN COMMAND on the AWS-UpdateSSMAgent document. It tries to upgrade to the latest version of amazon-ssm-agent but failed to do so. I tried to refresh the snap from 3.1.1188.0 but no updates available.. also tried to remove and reinstall the package but still get the 3.1.1188.0 version. My Windows Server instance get updated but all the Ubuntu 20.04LTS and 22.04LTS have issues with that update. I know 22.04 is not fully supported by SSM but the ssm agent update document always worked prior to the last 3 days. Anyone having similar issues? Here's the log: ``` Successfully downloaded manifest Successfully downloaded updater version 3.1.1446.0 Updating amazon-ssm-agent from 3.1.1188.0 to 3.1.1446.0 Successfully downloaded https://s3.ca-central-1.amazonaws.com/amazon-ssm-ca-central-1/amazon-ssm-agent/3.1.1188.0/amazon-ssm-agent-ubuntu-amd64.tar.gz Successfully downloaded https://s3.ca-central-1.amazonaws.com/amazon-ssm-ca-central-1/amazon-ssm-agent/3.1.1446.0/amazon-ssm-agent-ubuntu-amd64.tar.gz Initiating amazon-ssm-agent update to 3.1.1446.0 failed to install amazon-ssm-agent 3.1.1446.0, ErrorMessage=The execution of command returned Exit Status: 125 exit status 125 Initiating rollback amazon-ssm-agent to 3.1.1188.0 failed to install amazon-ssm-agent 3.1.1188.0, ErrorMessage=The execution of command returned Exit Status: 125 exit status 125 Failed to update amazon-ssm-agent to 3.1.1446.0 ```
2
answers
1
votes
312
views
asked 4 months ago

It there a way to add Exponential Backoff to AWS:ExecuteAutomation in an SSM Automation Document?

Put simply, I have written an SSM Automation. It creates a snapshot of each of the attached volumes of a targeted instance, by getting a list of Volume Ids from a DescribeInstance call. It then utilizes Rate Execution of a second runbook, in an AWS:executeAutomation action, which creates each snapshot, fanning out on those volume ids. As you can see, I have already limited MaxConcurrency of this step to 1. ``` { "name":"SnapshotAllVolumes", "action":"aws:executeAutomation", "maxAttempts": 3, "onFailure":"Abort", "inputs":{ "DocumentName": "MyCustomCreateSnapshotRunbook", "Targets": [ { "Key": "ParameterValues", "Values": [ "{{ DescribeInstance.CurrentVolumes }}" ] } ], "TargetParameterName": "VolumeId", "RuntimeParameters": { "InstanceId": "{{ InstanceId }}", "InstanceName": "{{ GetInstanceName.Name }}" }, "MaxConcurrency": "1" } ``` Where I have gotten into trouble is I am attempting to execute this runbook via Maintenance Window on ~60 instances, each averaging about 3 EBS volumes. I keep crashing into a rate limit on the above step. > Step fails when it is Executing. Fail to start automation, errorMessage: Rate exceeded. Please refer to Automation Service Troubleshooting Guide for more diagnosis details. Unfortunately, it doesn't tell me exactly which rate limit I'm hitting, but I think I can assume it is one of two: either the limit on api calls, or the limit on simultaneous SSM rate executions. Because the latter is much more restrictive(25 is the max), I think it's my most likely suspect. I've been dialing down the concurrency limit on my Maintenance Window. If my assumption about rate limit is correct, I need to stay under 25 concurrent Rate Executions. With the max concurrency of 1 on the child runbook, no individual execution of the parent runbook should lead to more than 2 simultaneous rate executions(the parent, and x*(number of volumes) consecutive executions). This means my concurrency rate for my maintenance window needs to keep below 25/2, so I've reckoned my max safe concurrency is a mere 12. Talking all this over with support, the solution recommended was to implement some sort of exponential backoff on the retries here. That way the calls that are rate limited on the first attempt are retried at different times, et al. This could be done through code if resorted to invoking a lambda that executed the child Automation, instead of executing directly in the Parent Runbook. I'd much rather not need to introduce a dependency on a Lambda here. I was wondering if anyone has a way to implement incremental backoff purely in SSM automation? Perhaps it isn't even possible?
1
answers
0
votes
45
views
asked 5 months ago

Unable to create new OpsItems from EventBridge when using Input Transformer for deduplication and adding category and severity values

Apologize to all for the duplicate post. I created my login under the wrong account when I initially posted this question. I’m able to generate a new OpsItem for any EC2, SecurityGroup, or VPC configuration change using an EventBridge rule with the following event pattern. { "source": "aws.config", "detail-type": "Config Configuration Item Change", "detail": { "messageType": "ConfigurationItemChangeNotification", "configurationItem": { "resourceType": "AWS::EC2::Instance", "AWS::EC2::SecurityGroup", "AWS::EC2::VPC" } } } The rule and target work great when using Matched event for the Input but I noticed that launching one EC2 using the AWS wizard creates at least three OpsItems, one for each resourceType. Therefore I’d like to implement a deduplication string to cut down on the number of OpsItems generated to one if possible and I’d also like to attach a category and severity to the new OpsItem. I’m trying to use an Input Transformer as recommended by the AWS documentation but even the most simplest of Input Transformers when applied prevent any new OpsItems from being generated. When I've tested, I've also ensured that all previous OpsItems were resolved. Can anyone tell me what might be blocking the creation of any new OpsItems when using this Input Transformer configuration? Here’s what I have configured now. Input path { "awsAccountId": "$.detail.configurationItem.awsAccountId", "awsRegion": "$.detail.configurationItem.awsRegion", "configurationItemCaptureTime": "$.detail.configurationItem.configurationItemCaptureTime", "detail-type": "$.detail-type", "messageType": "$.detail.messageType", "notificationCreationTime": "$.detail.notificationCreationTime", "region": "$.region", "resourceId": "$.detail.configurationItem.resourceId", "resourceType": "$.detail.configurationItem.resourceType", "resources": "$.resources", "source": "$.source", "time": "$.time" } Input template { "awsAccountId": "<awsAccountId>", "awsRegion": "<awsRegion>", "configurationItemCaptureTime": "<configurationItemCaptureTime>", "resourceId": "<resourceId>", "resourceType": "<resourceType>", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "<detail-type>", "source": "<source>", "time": "<time>", "region": "<region>", "resources": "<resources>", "messageType": "<messageType>", "notificationCreationTime": "<notificationCreationTime>", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{\"dedupString\":\"ConfigurationItemChangeNotification\"}" } } } Output when using the AWS supplied Sample event called “Config Configuration Item Change” { "awsAccountId": "123456789012", "awsRegion": "us-east-1", "configurationItemCaptureTime": "2022-03-16T01:10:50.837Z", "resourceId": "fs-01f0d526165b57f95", "resourceType": "AWS::EFS::FileSystem", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "Config Configuration Item Change", "source": "aws.config", "time": "2022-03-16T01:10:51Z", "region": "us-east-1", "resources": "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-01f0d526165b57f95", "messageType": "ConfigurationItemChangeNotification", "notificationCreationTime": "2022-03-16T01:10:51.976Z", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{"dedupString":"ConfigurationItemChangeNotification"}" } } }
1
answers
0
votes
85
views
asked 5 months ago

Unable to create new OpsItems from EventBridge when using Input Transformer for deduplication and adding category and severity values

I’m able to generate a new OpsItem for any EC2, SecurityGroup, or VPC configuration change using an EventBridge rule with the following event pattern. { "source": ["aws.config"], "detail-type": ["Config Configuration Item Change"], "detail": { "messageType": ["ConfigurationItemChangeNotification"], "configurationItem": { "resourceType": ["AWS::EC2::Instance", "AWS::EC2::SecurityGroup", "AWS::EC2::VPC"] } } } The rule and target work great when using Matched event for the Input but I noticed that launching one EC2 using the AWS wizard creates at least three OpsItems, one for each resourceType. Therefore I’d like to implement a deduplication string to cut down on the number of OpsItems generated to one if possible and I’d also like to attach a category and severity to the new OpsItem. I’m trying to use an Input Transformer as recommended by the AWS documentation but even the most simplest of Input Transformers when applied prevent any new OpsItems from being generated. When I've tested, I've also ensured that all previous OpsItems were resolved. Can anyone tell me what might be blocking the creation of any new OpsItems when using this Input Transformer configuration? Here’s what I have configured now. Input path { "awsAccountId": "$.detail.configurationItem.awsAccountId", "awsRegion": "$.detail.configurationItem.awsRegion", "configurationItemCaptureTime": "$.detail.configurationItem.configurationItemCaptureTime", "detail-type": "$.detail-type", "messageType": "$.detail.messageType", "notificationCreationTime": "$.detail.notificationCreationTime", "region": "$.region", "resourceId": "$.detail.configurationItem.resourceId", "resourceType": "$.detail.configurationItem.resourceType", "resources": "$.resources", "source": "$.source", "time": "$.time" } Input template { "awsAccountId": "<awsAccountId>", "awsRegion": "<awsRegion>", "configurationItemCaptureTime": "<configurationItemCaptureTime>", "resourceId": "<resourceId>", "resourceType": "<resourceType>", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "<detail-type>", "source": "<source>", "time": "<time>", "region": "<region>", "resources": "<resources>", "messageType": "<messageType>", "notificationCreationTime": "<notificationCreationTime>", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{\"dedupString\":\"ConfigurationItemChangeNotification\"}" } } } Output when using the AWS supplied Sample event called “Config Configuration Item Change” { "awsAccountId": "123456789012", "awsRegion": "us-east-1", "configurationItemCaptureTime": "2022-03-16T01:10:50.837Z", "resourceId": "fs-01f0d526165b57f95", "resourceType": "AWS::EFS::FileSystem", "title": "Template under ConfigDrift-EC2-Dedup4", "description": "Configuration Drift Detected.", "category": "Security", "severity": "3", "origination": "EventBridge Rule - ConfigDrift-EC2-Dedup", "detail-type": "Config Configuration Item Change", "source": "aws.config", "time": "2022-03-16T01:10:51Z", "region": "us-east-1", "resources": "arn:aws:elasticfilesystem:us-east-1:123456789012:file-system/fs-01f0d526165b57f95", "messageType": "ConfigurationItemChangeNotification", "notificationCreationTime": "2022-03-16T01:10:51.976Z", "operationalData": { "/aws/dedup": { "type": "SearchableString", "value": "{"dedupString":"ConfigurationItemChangeNotification"}" } } }
0
answers
0
votes
21
views
asked 5 months ago