CloudFormation Template failing while setting up Athena CUR
Hi AWS, I was querying Cost and Usage Reports (CURs) using Athena. By default a CloudFormation template gets created and it gets stored inside the bucket where the report is getting stored, however when I was trying to deploy the CloudFormation stack using the S3 Object URL CloudFormation template, it is failing with the error provided in the screenshot below.
The IAM user is having AdministratorAccess
and AWSGlueConsoleFullAccess
AWS managed IAM policies attached to it.
Here is the code for CloudFormation Template:
AWSTemplateFormatVersion: 2010-09-09
Resources:
AWSCURDatabase:
Type: 'AWS::Glue::Database'
Properties:
DatabaseInput:
Name: 'athenacurcfn_ec2_costing_report_cur'
CatalogId: !Ref AWS::AccountId
AWSCURCrawlerComponentFunction:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- glue.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
ManagedPolicyArns:
- !Sub 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSGlueServiceRole'
Policies:
- PolicyName: AWSCURCrawlerComponentFunction
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: !Sub 'arn:${AWS::Partition}:logs:*:*:*'
- Effect: Allow
Action:
- 'glue:UpdateDatabase'
- 'glue:UpdatePartition'
- 'glue:CreateTable'
- 'glue:UpdateTable'
- 'glue:ImportCatalogToGlue'
Resource: '*'
- Effect: Allow
Action:
- 's3:GetObject'
- 's3:PutObject'
Resource: !Sub 'arn:${AWS::Partition}:s3:::ec2-cur-reporting/ec2-cur/ec2-costing-report-cur/ec2-costing-report-cur*'
- PolicyName: AWSCURKMSDecryption
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'kms:Decrypt'
Resource: '*'
AWSCURCrawlerLambdaExecutor:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: AWSCURCrawlerLambdaExecutor
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: !Sub 'arn:${AWS::Partition}:logs:*:*:*'
- Effect: Allow
Action:
- 'glue:StartCrawler'
Resource: '*'
AWSCURCrawler:
Type: 'AWS::Glue::Crawler'
DependsOn:
- AWSCURDatabase
- AWSCURCrawlerComponentFunction
Properties:
Name: AWSCURCrawler-ec2-costing-report-cur
Description: A recurring crawler that keeps your CUR table in Athena up-to-date.
Role: !GetAtt AWSCURCrawlerComponentFunction.Arn
DatabaseName: !Ref AWSCURDatabase
Targets:
S3Targets:
- Path: 's3://ec2-cur-reporting/ec2-cur/ec2-costing-report-cur/ec2-costing-report-cur'
Exclusions:
- '**.json'
- '**.yml'
- '**.sql'
- '**.csv'
- '**.gz'
- '**.zip'
SchemaChangePolicy:
UpdateBehavior: UPDATE_IN_DATABASE
DeleteBehavior: DELETE_FROM_DATABASE
AWSCURInitializer:
Type: 'AWS::Lambda::Function'
DependsOn: AWSCURCrawler
Properties:
Code:
ZipFile: >
const { GlueClient, StartCrawlerCommand } = require('@aws-sdk/client-glue');
const response = require('./cfn-response');
exports.handler = function (event, context, callback) {
if (event.RequestType === 'Delete') {
response.send(event, context, response.SUCCESS);
} else {
const glue = new GlueClient();
const input = {
Name: 'AWSCURCrawler-ec2-costing-report-cur',
};
const command = new StartCrawlerCommand(input);
glue.send(command, function (err, data) {
if (err) {
const responseData = JSON.parse(this.httpResponse.body);
if (responseData['__type'] == 'CrawlerRunningException') {
callback(null, responseData.Message);
} else {
const responseString = JSON.stringify(responseData);
if (event.ResponseURL) {
response.send(event, context, response.FAILED, {
msg: responseString,
});
} else {
callback(responseString);
}
}
} else {
if (event.ResponseURL) {
response.send(event, context, response.SUCCESS);
} else {
callback(null, response.SUCCESS);
}
}
});
}
};
Handler: 'index.handler'
Timeout: 30
Runtime: nodejs18.x
ReservedConcurrentExecutions: 1
Role: !GetAtt AWSCURCrawlerLambdaExecutor.Arn
AWSStartCURCrawler:
Type: 'Custom::AWSStartCURCrawler'
Properties:
ServiceToken: !GetAtt AWSCURInitializer.Arn
AWSS3CUREventLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !GetAtt AWSCURInitializer.Arn
Principal: 's3.amazonaws.com'
SourceAccount: !Ref AWS::AccountId
SourceArn: !Sub 'arn:${AWS::Partition}:s3:::ec2-cur-reporting'
AWSS3CURLambdaExecutor:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: AWSS3CURLambdaExecutor
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: !Sub 'arn:${AWS::Partition}:logs:*:*:*'
- Effect: Allow
Action:
- 's3:PutBucketNotification'
Resource: !Sub 'arn:${AWS::Partition}:s3:::ec2-cur-reporting'
AWSS3CURNotification:
Type: 'AWS::Lambda::Function'
DependsOn:
- AWSCURInitializer
- AWSS3CUREventLambdaPermission
- AWSS3CURLambdaExecutor
Properties:
Code:
ZipFile: >
const { S3Client, PutBucketNotificationConfigurationCommand } = require('@aws-sdk/client-s3');
const response = require('./cfn-response');
exports.handler = function (event, context, callback) {
const s3 = new S3Client();
const putConfigRequest = function (notificationConfiguration) {
return new Promise(function (resolve, reject) {
const input = {
Bucket: event.ResourceProperties.BucketName,
NotificationConfiguration: notificationConfiguration,
};
const command = new PutBucketNotificationConfigurationCommand(input);
s3.send(command, function (err, data) {
if (err)
reject({
msg: this.httpResponse.body.toString(),
error: err,
data: data,
});
else resolve(data);
});
});
};
const newNotificationConfig = {};
if (event.RequestType !== 'Delete') {
newNotificationConfig.LambdaFunctionConfigurations = [
{
Events: ['s3:ObjectCreated:*'],
LambdaFunctionArn:
event.ResourceProperties.TargetLambdaArn || 'missing arn',
Filter: {
Key: {
FilterRules: [
{ Name: 'prefix', Value: event.ResourceProperties.ReportKey },
],
},
},
},
];
}
putConfigRequest(newNotificationConfig)
.then(function (result) {
response.send(event, context, response.SUCCESS, result);
callback(null, result);
})
.catch(function (error) {
response.send(event, context, response.FAILED, error);
console.log(error);
callback(error);
});
};
Handler: 'index.handler'
Timeout: 30
Runtime: nodejs18.x
ReservedConcurrentExecutions: 1
Role: !GetAtt AWSS3CURLambdaExecutor.Arn
AWSPutS3CURNotification:
Type: 'Custom::AWSPutS3CURNotification'
Properties:
ServiceToken: !GetAtt AWSS3CURNotification.Arn
TargetLambdaArn: !GetAtt AWSCURInitializer.Arn
BucketName: 'ec2-cur-reporting'
ReportKey: 'ec2-cur/ec2-costing-report-cur/ec2-costing-report-cur'
AWSCURReportStatusTable:
Type: 'AWS::Glue::Table'
DependsOn: AWSCURDatabase
Properties:
DatabaseName: athenacurcfn_ec2_costing_report_cur
CatalogId: !Ref AWS::AccountId
TableInput:
Name: 'cost_and_usage_data_status'
TableType: 'EXTERNAL_TABLE'
StorageDescriptor:
Columns:
- Name: status
Type: 'string'
InputFormat: 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OutputFormat: 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
SerdeInfo:
SerializationLibrary: 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
Location: 's3://ec2-cur-reporting/ec2-cur/ec2-costing-report-cur/cost_and_usage_data_status/'
Neither CloudFormation Linter nor ChatGPT detected any error in the infrastructure code. Please help as I have used the default generated template provided by AWS.
- 最新
- 投票最多
- 评论最多
Even if your IAM principal (user or role) is using the AdministratorAccess policy you may lack the necessary permissions to run the CFN template, if your account is part of an AWS Organization and the Organization has a Service Control Policies (SCP) defined that denies the action.
One way to troubleshoot is to use the Policy Simulator to see if your principal has the permissions to execute the CreateCrawler action for AWS Glue, since that's the step where your template is failing. If you find that your principal is not authorized to perform this action, and that your principal has the AdministratorAccess policy assigned, it is very likely that you have a SCP within your Organization that does not authorize this action. If that's the case, you'll need to work with your Organization administrators to modify the SCP to allow this action.
相关内容
- AWS 官方已更新 1 年前
- AWS 官方已更新 9 个月前
- AWS 官方已更新 9 个月前
Hi andrewsjw, it is a standalone account with no SCP applied. The weird part is the CloudFormation template is auto-generated by AWS, still it is erroring out. However, I have checked the user permission for the Glue action
CreateCrawler
and it is showing that it has the permission however I tried to deploy the template a couple of times but still that issue persists. Please help.I have attached an inline policy having
glue:CreateCrawler
access but I am getting the same error while deploying the CloudFormation template. Here is the error:Account XXXXXXXXXXXX is denied access. (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: f52b53db-4d10-4704-8e98-70ee46eb8f52; Proxy: null).
This is a little confusing because I have tried deploying the template using the root user as well but the same issue persists. These are the policies attached to the IAM user: