CUDOS Dashboard Updates keeps complaining about Lake Formation permissions.

Question

Post-deployment of the CUDOS Dashboards, if you want to migrate to CUR 2.0, you have to deploy the Data Exports stack as found below

https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi/migrate-to-cur2

and after updating the cloudformation stack for the dashboards, run a command line update in cloudshell.

That involves running the following,
`pip3 install -U git+https://github.com/aws-samples/aws-cudos-framework-deployment@cur-proxy`
`cid-cmd update --force --recursive`
then selecting selecting any one of the dashboards, and choosing `cid_data_export.cur2`. However, I have tried this many, many times, and after I choose `cid_data_export.cur2`, I get the following error,

` [cur-table-name-and-db] Please select CUR table: cid_data_export.cur2
        Athena table: cur2
        Resource IDs: yes
        SavingsPlans: yes
        Reserved Instances: yes

Analyzing view hourly_view
ERROR - Query:
CREATE OR REPLACE VIEW cid_tmp_deleteme AS
    SELECT DISTINCT
      "line_item_product_code" "product_code"
    , "product_servicecode" "service"
    , "line_item_operation" "operation"
    , "line_item_line_item_type" "charge_type"
    , "line_item_usage_type" "usage_type"
    , "line_item_line_item_description" "item_description"
    , "pricing_unit" "pricing_unit"
    , product['region'] "region"
    , "pricing_term" "pricing_term"
    , "bill_billing_period_start_date" "billing_period"
    , "line_item_usage_start_date" "usage_date"
    , "bill_payer_account_id" "payer_account_id"
    , "line_item_usage_account_id" "linked_account_id"
    , "savings_plan_savings_plan_a_r_n" "savings_plan_a_r_n"
    , "reservation_reservation_a_r_n" "reservation_a_r_n"
    , "sum"("line_item_unblended_cost") "unblended_cost"
    , "sum"("reservation_effective_cost") "reservation_effective_cost"
    , "sum"("savings_plan_savings_plan_effective_cost") "savings_plan_effective_cost"
    , "sum"("line_item_usage_amount") "usage_quantity"
    FROM
      "cid_data_export"."cur2"
    WHERE
      (((current_date - INTERVAL  '30' DAY) <= line_item_usage_start_date)
      AND ((("line_item_line_item_type" = 'Usage') OR ("line_item_line_item_type" = 'SavingsPlanCoveredUsage')) OR ("line_item_line_item_type" = 'DiscountedUsage'))
      AND "line_item_operation" NOT IN ('EKSPod-EC2','ECSTask-EC2'))
    GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15

Athena query status failed : Insufficient Lake Formation permission(s) on cid_tmp_deleteme (Service: AmazonDataCatalog; Status Code: 400; Error Code: AccessDeniedException; Request ID: ... Proxy: null)`

Firstly, my credentials are full admin permissions on everything. This is confirmed by running aws sts get-caller-identity on CloudShell. In terms of lake formation, I've given full permissions to the 	
cid_data_export database, and full permissions to the cur2 table. I've also tried giving full permissions to ALL tables of cid_data_export, and many different variations of this. I've tried giving full permission to the glue crawler, to the quicksight user, to CidQuickSightDataSourceRole, etc. I cannot figure out why it is complaining about permissions.

It seems that no matter what I do, it still keeps giving me the same Lake Formation error. I know I have the permissions, because I've confirmed this by manually copy + pasting + running the query in Athena, and it ran just fine. However, trying to do the mandatory update of the Dashboard through the CLI keeps giving Lake Formation permission issues.

If anybody has some insight into why this is happening, that would be incredibly useful. AWS Support has suggested it may be is trying to create the view in the wrong database (cur, rather than the data export stack), but then I wouldn't know why it would be trying to do that, the python package should do what it says on the tin, and nobody else seems to have complained about this - so I am wondering what's going wrong on my side.

I cannot skip this step as step 2.2 is mandatory:
https://catalog.workshops.aws/awscid/en-US/dashboards/foundational/cudos-cid-kpi/migrate-to-cur2

re:Post Agent · Answer

The issue you're experiencing with Lake Formation permissions while updating the CUDOS Dashboard is complex and can be caused by several factors. Let's break down the problem and potential solutions:

1. Lake Formation Permissions: Even though you have full admin permissions, Lake Formation uses a separate permission model. The error suggests that there might be an issue with permissions specifically for creating or replacing the view 'cid_tmp_deleteme'.

2. Database Location: The error occurs when trying to create a view named 'cid_tmp_deleteme'. It's possible that the CLI is attempting to create this view in a database where you don't have the necessary permissions, despite having access to the 'cid_data_export' database and 'cur2' table.

3. Granular Permissions: Lake Formation permissions can be very granular. You might need to ensure you have CREATE, ALTER, and DROP permissions for views in the specific database where 'cid_tmp_deleteme' is being created.

4. Role Assumptions: The CLI might be assuming a different role when executing these commands, which could have different permissions than your admin user.

5. Resource Links: Lake Formation uses resource links which could affect how permissions are applied across different databases or accounts.

To troubleshoot this issue:

1. Verify the exact database where 'cid_tmp_deleteme' is being created. You may need to grant permissions on this specific database in Lake Formation.

2. Ensure that the role or user executing the CLI commands has explicit permissions in Lake Formation to create views in the relevant database.

3. Check if there are any resource links in Lake Formation that might be affecting permission inheritance.

4. Try granting 'Super' permissions in Lake Formation to the executing role/user temporarily to see if it resolves the issue.

5. Verify that the Data Catalog settings in Lake Formation are not restricting view creation.

6. Check if there are any service-linked roles or resource-based policies that might be interfering with the permission model.

If the issue persists after trying these steps, you may need to engage with AWS support to investigate any potential bugs in the CUDOS framework deployment process or discrepancies between your account's Lake Formation setup and the expected configuration for the CUDOS dashboards.  
**Sources**  
[AWS::LakeFormation::Permissions - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lakeformation-permissions.html)  
[AWS::LakeFormation::PrincipalPermissions - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lakeformation-principalpermissions.html)  
[AWS::LakeFormation::Permissions DataLakePrincipal - AWS CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lakeformation-permissions-datalakeprincipal.html)

rePost-User-0573738 · Answer

Hello there,

Thank you for reaching out. This issue seems to be with the Lake formation permissions. Please refer the document - https://repost.aws/knowledge-center/glue-insufficient-lakeformation-permissions

If the issue persists after trying these steps, we would recommend you to engage with AWS support to investigate further.

I hope this helps, thank you!

CUDOS Dashboard Updates keeps complaining about Lake Formation permissions.

Relevant content