Bug with LoggingConfiguration

0

Hi,
We have been trying to enable LoggingConfiguration (Express/Standard) for the state functions we are developing as outlined by the below article:
https://aws.amazon.com/about-aws/whats-new/2020/02/aws-step-functions-supports-cloudwatch-logs-standard-workflows/
The configuration was working as expected over the last few months, but the same seems broken from the last week or so. When we try to add the LoggingConfiguration via CLI / Console / Cloud Formation, we get the below error:

The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID:XXXXXXXXXX)

Example CLI command:
aws stepfunctions update-state-machine --state-machine-arn <state machine ARN> --logging-configuration "{"level": "ALL", "includeExecutionData": true, "destinations": [{"cloudWatchLogsLogGroup": {"logGroupArn": "<log group ARN>"}}]}"

Cloud Formation:
ExampleValidatorStateMachine:
Type: AWS::StepFunctions::StateMachine
DependsOn: ExampleValidatorStateMachineLogGroup
Properties:
StateMachineName: !Sub '${StackPrefix}ExampleValidator'
LoggingConfiguration:
IncludeExecutionData: true
Level: 'ALL'
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn:
!GetAtt [ExampleValidatorStateMachineLogGroup, Arn]
DefinitionString: |-
{
…..
}
RoleArn: !GetAtt [ExampleValidatorStatesExecutionRole, Arn]

The StateMachine has associated role that has the required IAM Policies for Logging to CloudWatch Logs as outlined by:
https://docs.aws.amazon.com/step-functions/latest/dg/cw-logs.html

asked 4 years ago6834 views
8 Answers
1

Hello,

The issue is likely caused by a limitation of an internal dependency related to the size of the CloudWatch Logs resource policy. When you create a state machine with a new CloudWatch Log group, internally there is an update made on the resource policy document of the Cloudwatch Logs. If this policy document exceeds the 5120 character limit, you would see the error like this.

To avoid this issue, you can update the resource policy to use “Resource”: “*”.

Steps to do this:

  1. Use the AWS CLI to check the CloudWatch Logs resource policies:
    $ aws logs describe-resource-policies

  2. To update the policy:
    $ aws logs put-resource-policy —policy-name $POLICY_NAME —policy-document $POLICY_DOCUMENT

Where $POLICY_NAME is the name of the describe-resource-policies, usually in the form of AWSLogDeliveryWriteXXXX and $POLICY_DOCUMENT is a copy of the policyDocument from describe-resource-policies result with the Resource array replaced with “*”.

  1. Alternatively, you can remove unused entries from the Resource array, if you do not want to use a * policy (to have least privilege permissions)

I do understand this is manual work. Our service team has mentioned that they are working a long term fix for this issue. In the interim, kindly follow the work around to avoid access denied issues. Also, please verify the number of policies in the resource policy document in case the number exceeds 10 policies per document limit in which case, you may have to delete unused policies.

I hope this helps. Please let us know if you have any other queries.

answered 4 years ago
0

I also started seeing this error today when deploying a Cloudformation stack with logging configured on a state machine. It happens when deploying a new stack using our main branch (which has previously succeeeded) which makes it seem like something has changed.

Policy:
"MyPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": [
"logs:CreateLogDelivery",
"logs:GetLogDelivery",
"logs:UpdateLogDelivery",
"logs:DeleteLogDelivery",
"logs:ListLogDeliveries",
"logs:PutResourcePolicy",
"logs:DescribeResourcePolicies",
"logs:DescribeLogGroups"
],
"Effect": "Allow",
"Resource": "*"
},
...
],
"Version": "2012-10-17"
},
"PolicyName": "MyPolicy",
"Roles": [
{
"Ref": "MyRole"
}
]
}
},

Role:
"MyRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "states.xxx.amazonaws.com"
}
}
],
"Version": "2012-10-17"
}
},
},

State Machine:
"MyStateMachine": {
"Type": "AWS::StepFunctions::StateMachine",
"Properties": {
"RoleArn": {
"Fn::GetAtt": [
"MyRole",
"Arn"
]
},
"DefinitionString": ...,
"LoggingConfiguration": {
"Destinations": [
{
"CloudWatchLogsLogGroup": {
"LogGroupArn": {
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:xxx:xxx:log-group:",
{
"Fn::GetAtt": [
"MyLogGroup",
"LogGroupName"
]
},
":*"
]
]
}
}
}
],
"IncludeExecutionData": true,
"Level": "ALL"
},
"StateMachineName": "MyStateMachine"
},
"DependsOn": [
"MyPolicy",
"MyRole"
],
"UpdateReplacePolicy": "Retain",
},

The deployment fails with:
The state machine IAM Role is not authorized to access the Log Destination (Service: AWSStepFunctions; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxx; Proxy: null)

trigsby
answered 4 years ago
0

Thanks for the additional information. Were you able to resolve this issue using the workaround mentioned?

answered 4 years ago
0

As mentioned already, CloudWatch Log Groups have a limit on the size of the resource policy.

The permissions model was recently changed to grant AWS services access CloudWatch Logs if the log group name is prefixed by /aws/vendedlogs/ without increasing the size of the resource policy. The Step Functions console has been updated to create new log groups with the prefix /aws/vendedlogs/states/.

To find out more, please see the following documentation:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AWS-logs-and-resource-policy.html

AWS
wongada
answered 4 years ago
0

Changing the permissions/policy as follows worked for me:

  SomeStateMachine:  
    Type: AWS::Serverless::StateMachine  
      #...  
      Logging:  
        Destinations:  
          - CloudWatchLogsLogGroup:  
             LogGroupArn: !GetAtt StateMachineLogGroup.Arn  
        IncludeExecutionData: True  
        Level: ALL  
      Policies:  
        - CloudWatchLogsFullAccess  

So picking up an off the shelf I-do-anything-I-want policy, which (I'm guessing) side steps the issue of policy being crafted by CloudFormation/sam and stays below the policy size limit. So a workaround rather than a fix - over-permits, so use with caution and may not suit all corporate environments.

Edited by: johnskelton3 on Aug 26, 2020 4:31 AM

answered 4 years ago
0

I have tried to use "/aws/vendedlogs/states/" and it did not work but it worked for workaround fix "*" in policy resource array

answered 3 years ago
0

In my case, I exceeded the limit for the number of Cloudwatch Logs resource policies.

$ aws logs put-resource-policy --policy-name test --policy-document "{"Version":"2012-10-17","Statement":[{"Sid":"AWSLogDeliveryWrite","Effect":"Allow","Principal":{"Service":"delivery.logs.amazonaws.com"},"Action":["logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:aws:logs:us-east-1:164406550669:log-group:/aws/vendedlogs/states/data-cwdb-jbtest-step_function_pipeline_tf:log-stream:*"}]}"

An error occurred (LimitExceededException) when calling the PutResourcePolicy operation (reached max retries: 2): Resource limit exceeded.

Once I removed one of the existing resource policies, I was able to create a step function with the log configuration.

Edited by: blackburnja on May 11, 2021 11:55 AM

Edited by: blackburnja on May 11, 2021 11:56 AM

answered 3 years ago
0

In my case I have updated the log group resource policy to Resource array with "*". Still can't create/update step function with log enabled, either console or cloudformation, always saying "The state machine IAM Role is not authorized to access the Log Destination".

answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions